Setting up KVM to PXE boot virtual machines from a local TFTP server

I had occasion to test some failures that a partner is seeing in the field when trying to pxeboot the Ubuntu Server installer.  I’d never set up KVM to do this before, as I almost exclusively use MAAS to do all server deployments, and in fact, this is required for my work.  The partner is doing this as a side project and it seemed like a nice way to waste an afternoon.  The following is somewhat specific to my desktop running Ubuntu 18.04 LTS.  16.04 systems require a few extra steps to get a qemu that supports PXE booting.

On Ubuntu 16.04.x qemu-kvm from the Pike Ubuntu Cloud Archive (UCA) need to be installed.
The Pike Cloud Archive can be enabled like this:

$ sudo add-apt-repository cloud-archive:pike
$ sudo apt update

If you have 16.04 LTS,  after adding the cloud-archive repo and updating, proceed as you would for 17.04 and later:

$ sudo apt install -qq -y libvirt-daemon-system qemu-kvm virt-manager

Next, we need to get tftp installed and verify it’s running.

$ sudo apt install tftpd-hpa
$ sudo service tftpd-hpa status
● tftpd-hpa.service - LSB: HPA's tftp server
Loaded: loaded (/etc/init.d/tftpd-hpa; generated)
Active: active (running) since Thu 2018-09-13 09:35:49 EDT; 5s ago
Docs: man:systemd-sysv-generator(8)
Process: 16715 ExecStop=/etc/init.d/tftpd-hpa stop (code=exited, status=0/SUCC
Process: 16720 ExecStart=/etc/init.d/tftpd-hpa start (code=exited, status=0/SU
Tasks: 1 (limit: 4915)
CGroup: /system.slice/tftpd-hpa.service
└─16745 /usr/sbin/in.tftpd --listen --user tftp --address :69 --secur

Sep 13 09:35:49 galactica systemd[1]: Starting LSB: HPA's tftp server...
Sep 13 09:35:49 galactica tftpd-hpa[16720]: * Starting HPA's tftpd in.tftpd
Sep 13 09:35:49 galactica tftpd-hpa[16720]: ...done.
Sep 13 09:35:49 galactica systemd[1]: Started LSB: HPA's tftp server.

Now we need to start setting up the tftpboot directories.  I’m using /srv/tftp to host the files:

$ sudo mkdir /srv/tftp
$ cd /srv/tftp
$ $ sudo wget -nH -r --cut-dirs=8 http://archive.ubuntu.com/ubuntu/dists/bionic-updates/main/installer-amd64/current/images/netboot/

This will also leave a lot of cruft in the directory, perhaps that’s fixable with other wget options but this gets the files and directories you need.  You can clean up the cruft

$ sudo rm -f *.gif index.html* MANIFEST* MD5SUMS* SHA*
$ ls
boot.img.gz ldlinux.c32 pxelinux.0 udeb.list
debian-cd_info.tar.gz mini.iso pxelinux.cfg vmlinuz
initrd.gz netboot.tar.gz ubuntu-installer xen

Now we need to set permissions and ownership of the files:

/srv/tftp$ cd ../
/srv$ sudo chmod -R 777 tftp/
/srv$ sudo chown -R nobody: tftp/

Next, make sure TFTPD can see the right directory by pointing

$ cat /etc/default/tftpd-hpa
# /etc/default/tftpd-hpa

TFTP_USERNAME="tftp"
TFTP_DIRECTORY="/srv/tftp"
TFTP_ADDRESS=":69"
TFTP_OPTIONS="--secure --create"

Finally restart tftpd and verify it’s listening on the correct IP address (the gateway address for KVM’s bridge)

$ nc -uvz 192.168.123.1 69
Connection to 192.168.123.1 69 port [udp/tftp] succeeded!

This means that tftpd is active and listening on port 69 on my libvirt network (my ip addressing may be different from yours).  Now there’s a couple things to set for libvirt that will allow the VMs to grab the right boot files.  Add the lines (in bold) to the config:

$ virsh net-edit default
<network>
  <name>default</name>
  <uuid>c1ea51e9-ac51-455b-a6ff-7a222b6f94eb</uuid>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:42:de:cf'/>
  <ip address='192.168.123.1' netmask='255.255.255.0'>
    <tftp root='/srv/tftp'/>
    <dhcp>
      <range start='192.168.123.2' end='192.168.123.254'/>
      <bootp file='pxelinux.0' server='192.168.123.1'/>
    </dhcp>
  </ip>
</network>

Again, note that my IP addressing may be different.  Once the config has been modified to serve up PXE files, restart the virtual network:

$ virsh net-destroy default
Network default destroyed

$ virsh net-start default
Network default started

Now let’s create a VM to PXEBOOT the installer.  I’ll be using virt-manager for this, as when dealing with qemu, I prefer the simplicity of a GUI over remembering a large swath of command line arguments.