I have a number of computers that run Ubuntu 22.04 solely to act as a host for virtual machines I create and manage using virsh. The virsh command is just a wrapper around the libvirt package. In my case, I'm always using the KVM backend for libvirt. This takes advantage of the built-in support for virtualization on all modern 64-bit AMD & Intel CPUs.

One interesting feature available on newer CPUs is hugepages. I won't go into too much detail here, but using huge pages is primarily a technique to increase the effective of the TLB in the processor.

This explains how I enabled hugepages on my Ubuntu machines and configured libvirt to take advantage of them.

New tools & checking the state of the system

There are two new tools we'll use here which aren't commonly used on linux machines. Those are numastat and hugeadm. You can install those by running

sudo apt-get install libhugetlbfs-bin numactl

numastat

The numastat command shows the memory usage of a given process on the linux host. You can use it to find out how much memory & what kind of memory a process is using. For example we can run numastat -p $$

$ numastat -p $$

Per-node process memory usage (in MBs) for PID 2845190 (bash)
                           Node 0           Total
                  --------------- ---------------
Huge                         0.00            0.00
Heap                         1.52            1.52
Stack                        0.10            0.10
Private                      3.71            3.71
----------------  --------------- ---------------
Total                        5.33            5.33

This shows the memory usage for the shell proces. This isn't using much memory. The row for "Huge" is zero because huge pages have not been setup yet. I can find the memory usage for my virtual machine by running the same command but specifying the PID of the guest process in the host machine

$ sudo numastat -p 1746672

Per-node process memory usage (in MBs) for PID 1746672 (qemu-system-x86)
                           Node 0           Total
                  --------------- ---------------
Huge                         0.00            0.00
Heap                         8.34            8.34
Stack                        0.03            0.03
Private                   1101.44         1101.44
----------------  --------------- ---------------
Total                     1109.81         1109.81

Here we can see that a little over the configured 1 GB is being used by the VM.

hugeadm

The hugeadm tool is a command line tool to make the configuration of huge pages simpler. The first command to run is sudo hugeadm --page-sizes-all

$ sudo hugeadm --page-sizes-all
2097152
1073741824

This command lists the huge page sizes in bytes available. These correspond to 2 megabytes & 1 gigabyte, which are the sizes commonly available on x86-64 compatible hardware. Next we can run sudo hugeadm --pool-list to find out how many pages are configured

$ sudo hugeadm --pool-list
      Size  Minimum  Current  Maximum  Default
   2097152        0        0        0        *
1073741824        0        0        0 

Since this system is the default configuration, the quantity is for both sizes is zero.

Updating the configuration

I initially tried to use 1 GB huge pages for the virtual machines, but I discovered this does not work due to a bug in libvirt. This has since been fixed, but the fix is not present in the version of libvirt packaged into Ubuntu 22.04.

To update the kernel configuration to enable huge pages all we have to do is run something like hugeadm --pool-pages-max 2097152:100. This configures things so that 100 huge pages of 2 MB each is available. However, this does not persist after boot. So let's set this up to run at startup using systemd.

startup service

To use systemd to run a script on startup is simple. We first need to create a file at the path /etc/systemd/system/hugetlb-gigantic-pages.service. The contents should be as follows

[Unit]
Description=HugeTLB Gigantic Pages Reservation
DefaultDependencies=no
Before=dev-hugepages.mount
ConditionPathExists=/sys/devices/system/node

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/etc/systemd/hugetlb-reserve-pages.sh

[Install]
WantedBy=sysinit.target

This creates a service of type "oneshot" that runs at startup. It runs the script /etc/systemd/hugetlb-reserve-pages.sh. You can create that as follows

#!/bin/bash

set -xeuo pipefail
page_size=2097152
page_cnt=4096


hugeadm --pool-pages-max $page_size:$page_cnt
hugeadm --pool-pages-min $page_size:$page_cnt
hugeadm --pool-list

This bash script configures the minimum and maximum number of 2 MB huge pages to be 4096, which is 8 GB. My system has 16GB of memory and is mostly used for hosting virtual machines. So I am going to configure half the memory to be huge pages for now. Be sure and make the script executable by running sudo chmod a+x /etc/systemd/hugetlb-reserve-pages.sh. You can adjust the page_cnt variable if you want more or less memory to be setup for huge pages.

Next, tell systemd about the service by running sudo systemctl daemon-reload && sudo systemctl enable hugetlb-gigantic-pages. Now, restart the host machine. You can check that the startup script worked by running journalctl

$ sudo journalctl -u hugetlb-gigantic-pages
Jan 24 01:24:43 foo hugetlb-reserve-pages.sh[663]: + hugeadm --pool-pages-max 2097152:14336
Jan 24 01:24:43 foo hugetlb-reserve-pages.sh[663]: + hugeadm --pool-pages-min 2097152:14336
Jan 24 01:24:43 foo hugetlb-reserve-pages.sh[681]: hugeadm:WARNING: There is no swap space configured, resizing hugepage pool may fail
Jan 24 01:24:43 foo hugetlb-reserve-pages.sh[681]: hugeadm:WARNING: Use --add-temp-swap option to temporarily add swap during the resize
Jan 24 01:24:50 foo hugetlb-reserve-pages.sh[663]: + hugeadm --pool-list
Jan 24 01:24:50 foo hugetlb-reserve-pages.sh[986]:       Size  Minimum  Current  Maximum  Default
Jan 24 01:24:50 foo hugetlb-reserve-pages.sh[986]:    2097152    14336    14336    14336        *
Jan 24 01:24:50 foo hugetlb-reserve-pages.sh[986]: 1073741824        0        0        0
Jan 24 01:24:50 foo systemd[1]: Finished HugeTLB Gigantic Pages Reservation.

We can see that the script ran the two commands to update the number of pages & that afterwards the result is that the current number of 2 MB huge pages has been set.

You can download the two files I created above if you want to.

updating virtual machines to use hugepages

By default, virtual machines created using libvirt are not going to take advantage of those huge pages now enabled. This is changed by editing the XML for the host machine. In my case, my virtual machine is called "hass". First I shutdown the virtual machine. Then I run EDITOR=nano virsh edit hass. I can then insert a section that looks like this inside the XML document

<memoryBacking>
<hugepages/>
</memoryBacking>

Most virtual machines are not going to have an existing <memoryBacking> element. If they do, you can just put the <hugeages/> element under that. Save the file and close the editor. At this point the virtual machine is configured to use huge pages. I start it by by running virsh start hass in my case.

We can confirm this using numastat against the PID of the virtual machine

$ sudo numastat -p 1906

Per-node process memory usage (in MBs) for PID 1906 (qemu-system-x86)
                           Node 0           Total
                  --------------- ---------------
Huge                      1024.00         1024.00
Heap                         8.30            8.30
Stack                        0.04            0.04
Private                    105.23          105.23
----------------  --------------- ---------------
Total                     1137.56         1137.56

In this case the total amount of memory being used is the same, but the line for "Huge" indicates 1 gigabyte of memory is allocated from huge pages.