I was recently playing the video game Factorio which is a pretty cool independent video game. Anyone can host their own server on a linux host by downloading the executable and running it locally.

The game itself is very CPU intensive and as you add more user-created modifications to the game it tends to need even more CPU to keep up. I was running my game server on an Intel Core i3-8300T which has 4 physical CPU cores. The game only utilizes a single core. But with a quad core CPU, I surely should be able to reserve a single core for the game. This is probably the best way to get the most out of the existing hardware I have. This would be a bad idea for a program that is actually multithreaded, but turns out to be beneifical for single threaded programs.

Doing this requires at least 3 steps. Everything shown here has been tested on a Ubuntu 18.04 machine. This only works if your computer has multiple CPU cores, but almost every modern CPU comes with at least two cores.

Step 1: Set the process priority to the maximum

The first thing that needs to be changed is to give the process the highest scheduling priority. If you aren't familiar with scheduling priority, you can read more about it on Wikipedia. By setting the priority to the maximum value, you're telling the scheduler to put this process at the front of the line for the CPU access. The easy way to do this is

nice -n -19 my_command

The value -19 is the priority in this case. In linux, lower numbers indicate higher priority with -19 actually being the maximum priority. The problem with trying to do this is only the root user can set processes to this priority by default. If you try this now, here is what you'll see

$ nice -n -19 taskset 2 echo 'hello priority!'
nice: cannot set niceness: Permission denied
hello priority!

The nice command can't set the priority requested and shows an error message. To grant another user this permission, you'll need to edit /etc/security/limits.conf and add 3 more lines

myuser - nice -19
myuser soft nice -19
myuser hard nice -19

These 3 lines will allow myuser to set any process to the maximum priority. So replace myuser with the username you normally login to your linux box with. After that, you should be able to launch any process with maximum priority. To check this you can run the same command

$ nice -n -19 echo 'hello priority'
hello priority

If you still see the same error, log out and log back in.

Step 2: Set a taskset to only a single core

The next thing to do is to limit the process to only a single CPU core. This is done by setting a bitmask of the allowed CPU cores with the taskset command. I'll show the example here also with the nice command from the prior step.

$ nice -n -19 taskset 2 echo 'hello priority and taskset'

This should run the command with the taskset mask set as 2. What does 2 mean here? Well it's a bitmask. You can read more about what a bitmask is on Wikipedia. Each bit in the bitmask represents a single CPU Core. So if you have a quad core system you'd have bits like this when using a mask of 2

+-------------------------------------------------+
|CPU Core 3 | CPU Core 2 | CPU Core 1 | CPU Core 0|
+-------------------------------------------------+
|         0 |          0 |          1 |          0|
+-------------------------------------------------+

With a quad core CPU, only the lowest 4 bits matter. The value 2 has a binary representation of 10 with only one bit set. So this bitmask limits it to CPU Core #1. On most computers, all CPU cores are equal. For reasons I'll explain in the next section you shouldn't limit a process to CPU Core 0.

Step 3: Remove interrupts from the CPU core

In the previous section, we were able to launch a process at maximum priority and limited to a single CPU core. With maximum priority, it should have the entire core almost to itself. But processor interrupts can still run on this CPU. With a quad core CPU, restricting the interrupts to only 3 cores is an acceptable tradeoff. To do this, we need to set values in the /proc filesystem.

You can list out the interrupts by doing the following command

$ find /proc/irq/ -type f -name smp_affinity
/proc/irq/0/smp_affinity
/proc/irq/1/smp_affinity
/proc/irq/2/smp_affinity
/proc/irq/3/smp_affinity

If we examine one of these files, we can see what it contains

$ cat /proc/irq/131/smp_affinity
f

What does "f" mean here? Well, its the hexadecimal representation of a bitmask. The binary representation is just 1111, so four bits set. This bitmask works the same way as the bitmask from step 2, each bit specifies a CPU core that the interrupt may run on. Except here, we want to specify all of the CPU cores except the one we've reserved for the process. So since we set a value of 2 for the process, we want to set the inverse of that value. With 4 CPU cores, that bitmask is just the hexadecimal value d which has a binary representation of 1101. To apply this ti all interrupts, I ran the following command

$ find /proc/irq/ -type f -name smp_affinity | sudo xargs -L 1 -I '{}' /bin/bash -c 'echo {} && echo d > {}'
/proc/irq/0/smp_affinity
/bin/bash: line 0: echo: write error: Input/output error
/proc/irq/1/smp_affinity
/proc/irq/2/smp_affinity
/bin/bash: line 0: echo: write error: Input/output error
/proc/irq/3/smp_affinity
/proc/irq/4/smp_affinity
/proc/irq/5/smp_affinity
/proc/irq/6/smp_affinity
/proc/irq/7/smp_affinity
/proc/irq/8/smp_affinity

This runs a command like echo d > /proc/1/smp_affinity for each of the files. You'll notice some messages like "/bin/bash: line 0: echo: write error: Input/output error". There are two things I learned while doing this

1. Some interrupts can't have their affinity changed.
2. Some interupts have to be allowed to run on CPU Core 0.

So for the first problem, we'll just have to ignore it. Since some interrupts must be able to run on all CPU cores, we can't help that. But some interrupts seem to have to at least be allowed to run on CPU Core 0. This is why in Step 2 we set a bit mask of 2, reserving the process to CPU Core 1. The process is limited to Core 1, whereas the interrupts are mostly limited to the remaining cores.

The changes made to the smp_affinity files do not persist across reboots. If you want to apply this change on startup, my recommendation is to create a systemD service definition that runs a script. There is good example of that here.

Combining it all

Since all of the above is a bit much to remember, I put together a script that does everything. It takes a single CPU core (starting at 0) as its first argument, and the rest of the arguments are the command you want to run. Note this is only a good idea to use for 1 process per machine, as using this to run two processes at the same time results in the interrupt affinity being overwritten when the second process is launched. This script uses sudo, so it asks for your password if you're not already able to run as root.

#!/bin/bash

# Script to launch a process at maximum priority
# on a single specified cpu
# usage:
# ./reserve_cpu 2 my_command
set -e

cpu_count=$(grep processor /proc/cpuinfo | wc -l)
processor=$1
process_mask=$((1 << ${processor?}))
all_cpus=$((2 ** ${cpu_count?} -1))
irq_mask=$((${all_cpus?} ^ ${process_mask?}))

# This must be converted to hex
irq_mask=$(echo "obase=16; ${irq_mask?}" | bc)
set +e
find /proc/irq/ -type f -name smp_affinity | sudo xargs -L 1 -I '{}' /bin/bash -c "echo ${irq_mask?} > {} 2>/dev/null"
set -e

exec nice -n -19 taskset "${process_mask?}" ${@:2:99}

Example usage

$ ./reserve_cpu 2 echo 'I have a reservation'
I have a reservation