In order to perform some of the evaluations I write about on this blog, I have created a pretty automated setup around Virtualbox. This allows me to work pretty autonomously with my laptop with everything running locally. That’s pretty good for planes and trains, where accessing the cloud in a reliable way can be challenging.
Although this setup is not particularly tied to Kubernetes in any way, it does offer an automated Virtualbox setup for creating linked clones and setting hostnames easily. My particular setup is based around RHEL/CentOS, this setup could be pretty easily adapted to any other flavor of Linux too.
Overview of the process
There are two main parts to this; the first is a script to drive VirtualBox to create snapshots and linked clones from some base image, and to set properties that can be referenced from the guest OS.
The second part is a script run at system start that simply compares the current guest hostname to the Virtualbox property. If they do no match, then the hostname is updated and the system rebooted.
Creating the base VM
This is as simple as creating a normal guest VM and installing the OS on here. This is well documented elsewhere, so I’m going to assume this is done already. I named my base VM ‘playground-base’.
In case it’s not obvious, anything you set up in ‘playground-base’ will be available to all of the resulting linked clone VMs. I help myself out by adding my ssh key to the authorized_keys file of the ‘beyond’ user. I also create a key for that user and add it too. That allows my to ssh from my user on the host machine to beyond@playground*. I can also ssh between the playground VMs as the beyoud user.
I make sure that beyond can sudo NOPASSWD: as root. For the purpose of evaluating system level installations like Kubernetes, that makes things much easier.
Lastly, this whole procedure requires the Virtualbox Guest tools to be installed, so it’s a good idea to make sure that is done on the base image as well.
Add framework for the hostname update
For CentOS7 we are going to add a system unit that runs our script. This procedure should work for other flavors of Linux, but the file you need to update may change in that case
All of the scripts used here are available on github.
Create a file with the script somewhere you’ll remember. I simply added mine under /root
[beyond@playground-base ~]$ sudo cat /root/set-hostname
#!/bin/bash
required_hostname=$(VBoxControl guestproperty get /Startup/Hostname 2>&1 | grep "^Value" | sed 's/^Value: //')
current_hostname=$(hostname)
if [[ "X$required_hostname" != "X$current_hostname" ]]; then
if [[ "X$required_hostname" == "X" ]]; then
echo "No hostname specified in /Startup/Hostname property. Please set it"
sleep 600
else
echo "Hostname is incorrectly set. Setting to $required_hostname"
echo "$required_hostname" > /etc/hostname
sync
fi
reboot
fi
[beyond@playground-base ~]$
Now setup the unit file and enable the service:
[beyond@playground-base ~]$ cat >/etc/systemd/system/set-hostname.service <<EOF [Unit] Description=Set hostnmae from Virtualbox property After=network.target [Service] Type=simple ExecStart=/root/set-hostname TimeoutStartSec=0 [Install] WantedBy=default.target EOF [root@playground-base beyond]# systemctl daemon-reload [root@playground-base beyond]# systemctl list-unit-files | grep set-hostname set-hostname.service disabled [root@playground-base beyond]# systemctl enable set-hostname Created symlink from /etc/systemd/system/default.target.wants/set-hostname.service to /etc/systemd/system/set-hostname.service. [root@playground-base beyond]#
Before you actually reboot the machine, it is important to actually set the hostname property on the playground-base VM, otherwise it will sit sleeping and rebooting until that is done.
$ vboxmanage guestproperty set playground-base /Startup/Hostname playground-base
Now you can reboot the playground-base host and it should come back as normal.
Orchestrate the lab creation
The following simple bash script is what I use to create and destroy the ‘lab’ VMs. It starts by snapshotting the base and creating linked clones. It auto-starts the clones, which then change their hostnames and reboot.
#!/bin/bash
#
# Automatically create a test VM setup from a base image
# Creates linked clones to save space and anticipates
# that the hosts will set their hostname from the vbox
# property /Startup/Hostname
#
# See https://beyondthekube.com/my-lab-setup/ for details
#
# Known issues:
# - the sort -r for the delete of snapshots is a lttle
# lame. Works for <10 VMs, which suits my use-case
#
ACTION=$1
# Feel free to run as BASE_VM=my-better-base ./playground ..
BASE_VM="${BASE_VM:-playground-base}"
function usage() {
echo "Usage:"
echo "$(basename $0) (create|list|delete) [..options..]"
echo " create <number> [prefix]"
echo " list [prefix]"
echo " delete [prefix]"
exit 1
}
function snap_exists() {
# Return 0/1 if a VM snapshot of name $snap exists
local vm=$1
local snap=$2
vboxmanage snapshot ${vm} list 2>&1| grep "Name: ${snap} (" 2>&1
return $?
}
function vm_running() {
# Return 0/1 if a running vm matching $vm exists.
# exact math only unless $prefix is set true
local vm=$1
local prefix=$2
if [[ -z prefix ]]; then
vboxmanage list runningvms 2>&1 | grep "^\"${vm}\"" >/dev/null 2>&1
else
vboxmanage list runningvms 2>&1 | grep "^\"${vm}" >/dev/null 2>&1
fi
return $?
}
function list_vms() {
# list all VMS with prefix $prefix
local user_prefix=$1
local prefix="${user_prefix:-playground}"
# always exclude the $BASE_VM from lists
vboxmanage list vms | grep "^\"${prefix}" | grep -v "^\"${BASE_VM}\"" | awk '{print $1}' | sed 's/"//g'
}
function create_vms() {
# Create $num VMs from the $BASE_VM as linked clones
local num=$1
local user_prefix=$2
local prefix="${user_prefix:-playground}"
local clone
local snap
if vm_running $BASE_VM; then
echo "Cloning the base vm ${BASE_VM} requires it to be stopped. Please do that first"
fi
for i in $(seq 1 $num); do
clone="${prefix}${i}"
snap="${clone}-base"
if snap_exists ${BASE_VM} ${snap}; then
echo "Reusing existnig snapshot ${BASE_VM}::${snap}"
else
vboxmanage snapshot ${BASE_VM} take ${snap} --description "base snapshot for clone ${clone}"
fi
vboxmanage clonevm ${BASE_VM} --name ${clone} --snapshot ${snap} --options link --register
vboxmanage guestproperty set ${clone} /Startup/Hostname "$clone"
vboxmanage startvm ${clone}
done
}
function destroy_vms() {
# Delete VMs patching $prefix and associated snapshots
local prefix=$1
local snap
local snap_uuid
local gone
for vm in $(list_vms $prefix | sort -r); do
vboxmanage controlvm ${vm} poweroff
gone=1
while [[ $gone != 0 ]]; do
vboxmanage unregistervm ${vm} --delete >/dev/null 2>&1
gone=$?
done
snap="${vm}-base"
snap_uuid=$(snap_exists ${BASE_VM} ${snap} | sed 's/^.*UUID: \(.*\)).*/\1/')
while [[ ! -z ${snap_uuid} ]]; do
vboxmanage snapshot ${BASE_VM} delete ${snap_uuid}
sleep 1
snap_uuid=$(snap_exists ${BASE_VM} ${snap} | sed 's/^.*UUID: \(.*\)).*/\1/')
done
done
}
# Poor-man's argparsing
case "${ACTION}" in
"create")
shift
num=$1; shift
prefix=$1; shift
if [[ -z ${num} ]]; then
usage
fi
create_vms ${num} ${prefix}
;;
"list")
shift
prefix=$1; shift
list_vms ${prefix}
;;
"delete")
shift
prefix=$1; shift
destroy_vms ${prefix}
;;
*)
usage
;;
esac
An example session
The following shows a simple setup and teardown of a ‘lab.’ Note that in this example, the DNS setup is provided by my pfsense gateway which gets the hostnames from their boot-time DHCP query. An alternative setup would me to use sudo in the script to manage hosts, or write a function to lookup the hostname->ip using vbox manage and implement something like playground connect <hostname>'
~ $ playground create 3 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100% Snapshot taken. UUID: 7a641e8a-4458-4bb9-915d-02ec9ec4c20b 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100% Machine has been successfully cloned as "playground1" Waiting for VM "playground1" to power on... VM "playground1" has been successfully started. 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100% Snapshot taken. UUID: 491d5021-49c2-4fe3-a0c9-c0926b8a2cc6 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100% Machine has been successfully cloned as "playground2" Waiting for VM "playground2" to power on... VM "playground2" has been successfully started. 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100% Snapshot taken. UUID: 7592de93-1193-4426-a425-aa592736ad60 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100% Machine has been successfully cloned as "playground3" Waiting for VM "playground3" to power on... VM "playground3" has been successfully started. ~ $
I wait for those hosts to boot. I tend to run the VMs normally, as opposed to headless, so I can see this happen. This also allows me to recover the output of kernel panics, or otherwise debug more easily.
Now the lab is up and running and I can interact with any of the hosts as normal:
~ $ for i in 1 2 3; do ssh -o StrictHostKeyChecking=no beyond@playground$i uptime; done Warning: Permanently added 'playground1,192.168.1.166' (ECDSA) to the list of known hosts. 16:13:20 up 2 min, 0 users, load average: 0.10, 0.14, 0.06 Warning: Permanently added 'playground2,192.168.1.167' (ECDSA) to the list of known hosts. 16:13:20 up 2 min, 0 users, load average: 0.01, 0.01, 0.01 Warning: Permanently added 'playground3,192.168.1.168' (ECDSA) to the list of known hosts. 16:13:20 up 2 min, 0 users, load average: 0.10, 0.14, 0.06 ~ $
Tearing it down
When done, I can easily tear down the whole experiment:
~ $ playground delete 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100% Deleting snapshot 'playground3-base' (7592de93-1193-4426-a425-aa592736ad60) 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100% 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100% Deleting snapshot 'playground2-base' (491d5021-49c2-4fe3-a0c9-c0926b8a2cc6) 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100% 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100% Deleting snapshot 'playground1-base' (7a641e8a-4458-4bb9-915d-02ec9ec4c20b) 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100% ~ $
Additional features
The playground script supports an optional prefix argument, so it’s possible to run a different set of VMs with other prefixes. This can be useful if you are running a number of different labs on the same host.
You can also list the hosts associated with the lab by issuing playground list <optional prefix>
Conclusion
For the short time it took to figure this out, I believe this is a pretty interesting automation. It takes about 5-10 minutes to go through these steps manually via the GUI, and has a lot of easily typo’d steps.
Using these scripts it’s possible to build a fully working lab in < 1m, and tear it down even faster. I look forward to seeing if this actually changes my workflow in terms of evaluating different setups concurrently.