Configure CPU isolation for an instance

TechPreview

Note

Consider this section as part of Deploy an OpenStack cluster.

Warning

The below procedure applies only to deployments based on deprecated Ubuntu 20.04. For Ubuntu 22.04 that supports cgroup v2, use the cpushield module. For the procedure details, see Day-2 operations.

CPU isolation is a way to force the system scheduler to use only some logical CPU cores for processes. For compute hosts, you should typically isolate system processes and virtual guests on different cores through the cpusets mechanism in Linux kernel.

The Linux kernel and cpuset provide a mechanism to run tasks by limiting the resources defined by a cpuset. The tasks can be moved from one cpuset to another to use the resources defined in other cpusets. The cset Python tool is a command-line interface to work with cpusets.

To configure CPU isolation using cpusets:

1. Configure core isolation:

   Note

   You can also automate this step during deployment by using the postDeploy script as described in Create MOSK host profiles.

   cat <<-"EOF" > /usr/bin/setup-cgroups.sh
   #!/bin/bash
   
   set -x
   
   UNSHIELDED_CPUS=${UNSHIELDED_CPUS:-"0-3"}
   UNSHIELDED_MEM_NUMAS=${UNSHIELDED_MEM_NUMAS:-0}
   SHIELD_CPUS=${SHIELD_CPUS:-"4-15"}
   SHIELD_MODE=${SHIELD_MODE:-"cpuset"} # One of isolcpu or cpuset
   
   DOCKER_CPUS=${DOCKER_CPUS:-$UNSHIELDED_CPUS}
   DOCKER_MEM_NUMAS=${DOCKER_MEM_NUMAS:-$UNSHIELDED_MEM_NUMAS}
   KUBERNETES_CPUS=${KUBERNETES_CPUS:-$UNSHIELDED_CPUS}
   KUBERNETES_MEM_NUMAS=${KUBERNETES_MEM_NUMAS:-$UNSHIELDED_MEM_NUMAS}
   CSET_CMD=${CSET_CMD:-"python3 /usr/bin/cset"}
   
   if [[ ${SHIELD_MODE} == "cpuset" ]]; then
       ${CSET_CMD} set -c ${UNSHIELDED_CPUS} -m ${UNSHIELDED_MEM_NUMAS} -s system
       ${CSET_CMD} proc -m -f root -t system
       ${CSET_CMD} proc -k -f root -t system
   fi
   
   ${CSET_CMD} set --cpu=${DOCKER_CPUS} --mem=${DOCKER_MEM_NUMAS} --set=docker
   ${CSET_CMD} set --cpu=${KUBERNETES_CPUS} --mem=${KUBERNETES_MEM_NUMAS} --set=kubepods
   ${CSET_CMD} set --cpu=${DOCKER_CPUS} --mem=${DOCKER_MEM_NUMAS} --set=com.docker.ucp
   
   EOF
   chmod +x /usr/bin/setup-cgroups.sh
   
   cat <<-"EOF" > /etc/systemd/system/shield-cpus.service
   [Unit]
   Description=Shield CPUs
   DefaultDependencies=no
   After=systemd-udev-settle.service
   Before=lvm2-activation-early.service
   Wants=systemd-udev-settle.service
   [Service]
   ExecStart=/usr/bin/setup-cgroups.sh
   RemainAfterExit=true
   Type=oneshot
   Restart=on-failure     #Service should restart on failure
   RestartSec=5s          #Restart each five seconds until success
   [Install]
   WantedBy=basic.target
   EOF
   
   systemctl enable shield-cpus
   
   reboot
   

2. As root user, verify that isolation has been applied:

   cset set -l
   

   Example of system response:

   cset:
         Name       CPUs-X     MEMs-X    Tasks Subs   Path
     ------------ ---------- - ------- - ----- ---- ----------
     root             0-15 y       0 y     165    4  /
     kubepods         0-3 n        0 n       0    2  /kubepods
     docker           0-3 n        0 n       0    0  /docker
     system           0-3 n        0 n      65    0  /system
     com.docker.ucp   0-3 n        0 n       0    0  /com.docker.ucp
   

3. Run the cpustress container:

   docker run -it --name cpustress --rm containerstack/cpustress --cpu 4 --timeout 30s --metrics-brief
   

4. Verify that isolated cores are not affected:

   htop
   

   Example of system response highlighting the load created on all available Docker cores: