Using hugepages with libvirt on Ubuntu 22.04
- Monday January 23 2023
- linux virtualization
I have a number of computers that run Ubuntu 22.04 solely to act as a host for virtual machines I create and manage using virsh
. The virsh
command is just a wrapper around the libvirt
package. In my case, I'm always using the KVM backend for libvirt
. This takes advantage of the built-in support for virtualization on all modern 64-bit AMD & Intel CPUs.
One interesting feature available on newer CPUs is hugepages. I won't go into too much detail here, but using huge pages is primarily a technique to increase the effective of the TLB in the processor.
This explains how I enabled hugepages on my Ubuntu machines and configured libvirt
to take advantage of them.
New tools & checking the state of the system
There are two new tools we'll use here which aren't commonly used on linux machines. Those are numastat
and hugeadm
. You can install those by running
sudo apt-get install libhugetlbfs-bin numactl
numastat
The numastat
command shows the memory usage of a given process on the linux host. You can use it to find out how much memory & what kind of memory a process is using. For example we can run numastat -p $$
$ numastat -p $$ Per-node process memory usage (in MBs) for PID 2845190 (bash) Node 0 Total --------------- --------------- Huge 0.00 0.00 Heap 1.52 1.52 Stack 0.10 0.10 Private 3.71 3.71 ---------------- --------------- --------------- Total 5.33 5.33
This shows the memory usage for the shell proces. This isn't using much memory. The row for "Huge" is zero because huge pages have not been setup yet. I can find the memory usage for my virtual machine by running the same command but specifying the PID of the guest process in the host machine
$ sudo numastat -p 1746672 Per-node process memory usage (in MBs) for PID 1746672 (qemu-system-x86) Node 0 Total --------------- --------------- Huge 0.00 0.00 Heap 8.34 8.34 Stack 0.03 0.03 Private 1101.44 1101.44 ---------------- --------------- --------------- Total 1109.81 1109.81
Here we can see that a little over the configured 1 GB is being used by the VM.
hugeadm
The hugeadm
tool is a command line tool to make the configuration of huge pages simpler. The first command to run is sudo hugeadm --page-sizes-all
$ sudo hugeadm --page-sizes-all 2097152 1073741824
This command lists the huge page sizes in bytes available. These correspond to 2 megabytes & 1 gigabyte, which are the sizes commonly available on x86-64 compatible hardware. Next we can run sudo hugeadm --pool-list
to find out how many pages are configured
$ sudo hugeadm --pool-list Size Minimum Current Maximum Default 2097152 0 0 0 * 1073741824 0 0 0
Since this system is the default configuration, the quantity is for both sizes is zero.
Updating the configuration
I initially tried to use 1 GB huge pages for the virtual machines, but I discovered this does not work due to a bug in libvirt. This has since been fixed, but the fix is not present in the version of libvirt
packaged into Ubuntu 22.04.
To update the kernel configuration to enable huge pages all we have to do is run something like hugeadm --pool-pages-max 2097152:100
. This configures things so that 100 huge pages of 2 MB each is available. However, this does not persist after boot. So let's set this up to run at startup using systemd.
startup service
To use systemd to run a script on startup is simple. We first need to create a file at the path /etc/systemd/system/hugetlb-gigantic-pages.service
. The contents should be as follows
[Unit] Description=HugeTLB Gigantic Pages Reservation DefaultDependencies=no Before=dev-hugepages.mount ConditionPathExists=/sys/devices/system/node [Service] Type=oneshot RemainAfterExit=yes ExecStart=/etc/systemd/hugetlb-reserve-pages.sh [Install] WantedBy=sysinit.target
This creates a service of type "oneshot" that runs at startup. It runs the script /etc/systemd/hugetlb-reserve-pages.sh
. You can create that as follows
#!/bin/bash set -xeuo pipefail page_size=2097152 page_cnt=4096 hugeadm --pool-pages-max $page_size:$page_cnt hugeadm --pool-pages-min $page_size:$page_cnt hugeadm --pool-list
This bash script configures the minimum and maximum number of 2 MB huge pages to be 4096, which is 8 GB. My system has 16GB of memory and is mostly used for hosting virtual machines. So I am going to configure half the memory to be huge pages for now. Be sure and make the script executable by running sudo chmod a+x /etc/systemd/hugetlb-reserve-pages.sh
. You can adjust the page_cnt
variable if you want more or less memory to be setup for huge pages.
Next, tell systemd about the service by running sudo systemctl daemon-reload && sudo systemctl enable hugetlb-gigantic-pages
. Now, restart the host machine. You can check that the startup script worked by running journalctl
$ sudo journalctl -u hugetlb-gigantic-pages Jan 24 01:24:43 foo hugetlb-reserve-pages.sh[663]: + hugeadm --pool-pages-max 2097152:14336 Jan 24 01:24:43 foo hugetlb-reserve-pages.sh[663]: + hugeadm --pool-pages-min 2097152:14336 Jan 24 01:24:43 foo hugetlb-reserve-pages.sh[681]: hugeadm:WARNING: There is no swap space configured, resizing hugepage pool may fail Jan 24 01:24:43 foo hugetlb-reserve-pages.sh[681]: hugeadm:WARNING: Use --add-temp-swap option to temporarily add swap during the resize Jan 24 01:24:50 foo hugetlb-reserve-pages.sh[663]: + hugeadm --pool-list Jan 24 01:24:50 foo hugetlb-reserve-pages.sh[986]: Size Minimum Current Maximum Default Jan 24 01:24:50 foo hugetlb-reserve-pages.sh[986]: 2097152 14336 14336 14336 * Jan 24 01:24:50 foo hugetlb-reserve-pages.sh[986]: 1073741824 0 0 0 Jan 24 01:24:50 foo systemd[1]: Finished HugeTLB Gigantic Pages Reservation.
We can see that the script ran the two commands to update the number of pages & that afterwards the result is that the current number of 2 MB huge pages has been set.
You can download the two files I created above if you want to.
updating virtual machines to use hugepages
By default, virtual machines created using libvirt
are not going to take advantage of those huge pages now enabled. This is changed by editing the XML for the host machine. In my case, my virtual machine is called "hass". First I shutdown the virtual machine. Then I run EDITOR=nano virsh edit hass
. I can then insert a section that looks like this inside the XML document
<memoryBacking> <hugepages/> </memoryBacking>
Most virtual machines are not going to have an existing <memoryBacking>
element. If they do, you can just put the <hugeages/>
element under that. Save the file and close the editor. At this point the virtual machine is configured to use huge pages. I start it by by running virsh start hass
in my case.
We can confirm this using numastat
against the PID of the virtual machine
$ sudo numastat -p 1906 Per-node process memory usage (in MBs) for PID 1906 (qemu-system-x86) Node 0 Total --------------- --------------- Huge 1024.00 1024.00 Heap 8.30 8.30 Stack 0.04 0.04 Private 105.23 105.23 ---------------- --------------- --------------- Total 1137.56 1137.56
In this case the total amount of memory being used is the same, but the line for "Huge" indicates 1 gigabyte of memory is allocated from huge pages.