Configure CPU isolation for an instance¶

Note

Consider this section as part of Deploy an OpenStack cluster.

CPU isolation is a way to force the system scheduler to use only some logical CPU cores for processes. For compute hosts, you should typically isolate system processes and virtual guests on different cores. This section describes the two possible options on how to achieve this:

Through the isolcpus configuration parameter for Linux kernel ^{Deprecated since
MOSK 22.2}
Through the cpusets mechanism in Linux kernel ^{Available since MOSK 22.2, TechPreview}

For details, see OpenStack official documentation: CPU topologies and Shielding Linux Resources.

Configure CPU isolation using isolcpus¶

Note

Starting from MOSK 22.2, isolcpus is deprecated.

Using the isolcpus parameter, specific CPUs are removed from the general kernel symmetrical multiprocessing (SMP) load balancing and scheduling. The only way to get tasks scheduled onto isolated CPUs is taskset. The list of isolcpus is configured statically at boot time. You can only change it by rebooting with a different value. In Linux kernel, the isolcpus parameter is deprecated in favor of cpusets.

To configure CPU isolation using isolcpus:

Configure isolcpus in Linux kernel:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=4-15"

Apply the changes:
```
update-grub
```

Isolate cores from scheduling of Docker or Kubernetes workloads:

cat <<-"EOF" > /usr/bin/setup-cgroups.sh
#!/bin/bash

set -x

UNSHIELDED_CPUS=${UNSHIELDED_CPUS:-"0-3"}
SHIELD_CPUS=${SHIELD_CPUS:-"4-15"}
SHIELD_MODE=${SHIELD_MODE:-"isolcpu"} # One of isolcpu or cpuset

DOCKER_CPUS=${DOCKER_CPUS:-$UNSHIELDED_CPUS}
KUBERNETES_CPUS=${KUBERNETES_CPUS:-$UNSHIELDED_CPUS}
CSET_CMD=${CSET_CMD:-"python2 /usr/bin/cset"}

if [[ ${SHIELD_MODE} == "cpuset" ]]; then
    ${CSET_CMD} set -c ${UNSHIELDED_CPUS} -s system
    ${CSET_CMD} proc -m -f root -t system
    ${CSET_CMD} proc -k -f root -t system
fi

${CSET_CMD} set --cpu=${DOCKER_CPUS} --set=docker
${CSET_CMD} set --cpu=${KUBERNETES_CPUS} --set=kubepods
${CSET_CMD} set --cpu=${DOCKER_CPUS} --set=com.docker.ucp

EOF
chmod +x /usr/bin/setup-cgroups.sh

cat <<-"EOF" > /etc/systemd/system/shield-cpus.service
[Unit]
Descriptio=Shield CPUs
DefaultDependencies=no
After=systemd-udev-settle.service
Before=lvm2-activation-early.service
Wants=systemd-udev-settle.service
[Service]
ExecStart=/usr/bin/setup-cgroups.sh
RemainAfterExit=true
Type=oneshot
[Install]
WantedBy=basic.target
EOF

systemctl enable shield-cpus

As root user, reboot the host:
```
cat /sys/devices/system/cpu/isolated
```
Example of system response:
```
4-15
```

As root user, verify that isolation is active:

cset set -l

Example of system response:

cset:
     Name       CPUs-X      MEMs-X   Tasks Subs   Path
 ------------ ---------- - ------- - ----- ---- ----------
 root             0-15 y       0 y    1449    3  /
 kubepods         0-3 n        0 n       0    2  /kubepods
 docker           0-3 n        0 n       0    5  /docker
 com.docker.ucp   0-3 n        0 n       0    1  /com.docker.ucp

Configure CPU isolation using cpusets¶

Available since MOSK 22.2 TechPreview

The Linux kernel and cpuset provide a mechanism to run tasks by limiting the resources defined by a cpuset. The tasks can be moved from one cpuset to another to use the resources defined in other cpusets. The cset Python tool is a command-line interface to work with cpusets.

To configure CPU isolation using cpusets:

Configure core isolation:

Note

You can also automate this step during deployment by using the postDeploy script as described in Create MOSK host profiles.

cat <<-"EOF" > /usr/bin/setup-cgroups.sh
#!/bin/bash

set -x

UNSHIELDED_CPUS=${UNSHIELDED_CPUS:-"0-3"}
SHIELD_CPUS=${SHIELD_CPUS:-"4-15"}
SHIELD_MODE=${SHIELD_MODE:-"cpuset"} # One of isolcpu or cpuset

DOCKER_CPUS=${DOCKER_CPUS:-$UNSHIELDED_CPUS}
KUBERNETES_CPUS=${KUBERNETES_CPUS:-$UNSHIELDED_CPUS}
CSET_CMD=${CSET_CMD:-"python2 /usr/bin/cset"}

if [[ ${SHIELD_MODE} == "cpuset" ]]; then
    ${CSET_CMD} set -c ${UNSHIELDED_CPUS} -s system
    ${CSET_CMD} proc -m -f root -t system
    ${CSET_CMD} proc -k -f root -t system
fi

${CSET_CMD} set --cpu=${DOCKER_CPUS} --set=docker
${CSET_CMD} set --cpu=${KUBERNETES_CPUS} --set=kubepods
${CSET_CMD} set --cpu=${DOCKER_CPUS} --set=com.docker.ucp

EOF
chmod +x /usr/bin/setup-cgroups.sh

cat <<-"EOF" > /etc/systemd/system/shield-cpus.service
[Unit]
Description=Shield CPUs
DefaultDependencies=no
After=systemd-udev-settle.service
Before=lvm2-activation-early.service
Wants=systemd-udev-settle.service
[Service]
ExecStart=/usr/bin/setup-cgroups.sh
RemainAfterExit=true
Type=oneshot
Restart=on-failure     #Service should restart on failure
RestartSec=5s          #Restart each five seconds until success
[Install]
WantedBy=basic.target
EOF

systemctl enable shield-cpus

reboot

As root user, verify that isolation has been applied:

cset set -l

Example of system response:

cset:
      Name       CPUs-X     MEMs-X    Tasks Subs   Path
  ------------ ---------- - ------- - ----- ---- ----------
  root             0-15 y       0 y     165    4  /
  kubepods         0-3 n        0 n       0    2  /kubepods
  docker           0-3 n        0 n       0    0  /docker
  system           0-3 n        0 n      65    0  /system
  com.docker.ucp   0-3 n        0 n       0    0  /com.docker.ucp

Run the cpustress container:

docker run -it --name cpustress --rm containerstack/cpustress --cpu 4 --timeout 30s --metrics-brief

Verify that isolated cores are not affected:
```
htop
```
Example of system response highlighting the load created on all available Docker cores: