Configure CPU isolation for an instance¶
Note
Consider this section as part of Deploy an OpenStack cluster.
CPU isolation is a way to force the system scheduler to use only some logical CPU cores for processes. For compute hosts, you should typically isolate system processes and virtual guests on different cores. This section describes the two possible options on how to achieve this:
Through the
isolcpus
configuration parameter for Linux kernel Deprecated since MOSK 22.2Through the
cpusets
mechanism in Linux kernel Available since MOSK 22.2, TechPreview
For details, see OpenStack official documentation: CPU topologies and Shielding Linux Resources.
Configure CPU isolation using isolcpus¶
Note
Starting from MOSK 22.2, isolcpus
is
deprecated.
Using the isolcpus
parameter, specific CPUs are removed from the general
kernel symmetrical multiprocessing (SMP) load balancing and scheduling. The
only way to get tasks scheduled onto isolated CPUs is taskset
. The list of
isolcpus
is configured statically at boot time. You can only change it by
rebooting with a different value. In Linux kernel, the isolcpus
parameter is deprecated in favor of cpusets.
To configure CPU isolation using isolcpus:
Configure
isolcpus
in Linux kernel:GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=4-15"
Apply the changes:
update-grub
Isolate cores from scheduling of Docker or Kubernetes workloads:
cat <<-"EOF" > /usr/bin/setup-cgroups.sh #!/bin/bash set -x UNSHIELDED_CPUS=${UNSHIELDED_CPUS:-"0-3"} SHIELD_CPUS=${SHIELD_CPUS:-"4-15"} SHIELD_MODE=${SHIELD_MODE:-"isolcpu"} # One of isolcpu or cpuset DOCKER_CPUS=${DOCKER_CPUS:-$UNSHIELDED_CPUS} KUBERNETES_CPUS=${KUBERNETES_CPUS:-$UNSHIELDED_CPUS} CSET_CMD=${CSET_CMD:-"python2 /usr/bin/cset"} if [[ ${SHIELD_MODE} == "cpuset" ]]; then ${CSET_CMD} set -c ${UNSHIELDED_CPUS} -s system ${CSET_CMD} proc -m -f root -t system ${CSET_CMD} proc -k -f root -t system fi ${CSET_CMD} set --cpu=${DOCKER_CPUS} --set=docker ${CSET_CMD} set --cpu=${KUBERNETES_CPUS} --set=kubepods ${CSET_CMD} set --cpu=${DOCKER_CPUS} --set=com.docker.ucp EOF chmod +x /usr/bin/setup-cgroups.sh cat <<-"EOF" > /etc/systemd/system/shield-cpus.service [Unit] Descriptio=Shield CPUs DefaultDependencies=no After=systemd-udev-settle.service Before=lvm2-activation-early.service Wants=systemd-udev-settle.service [Service] ExecStart=/usr/bin/setup-cgroups.sh RemainAfterExit=true Type=oneshot [Install] WantedBy=basic.target EOF systemctl enable shield-cpus
As root user, reboot the host:
cat /sys/devices/system/cpu/isolated
Example of system response:
4-15
As root user, verify that isolation is active:
cset set -l
Example of system response:
cset: Name CPUs-X MEMs-X Tasks Subs Path ------------ ---------- - ------- - ----- ---- ---------- root 0-15 y 0 y 1449 3 / kubepods 0-3 n 0 n 0 2 /kubepods docker 0-3 n 0 n 0 5 /docker com.docker.ucp 0-3 n 0 n 0 1 /com.docker.ucp
Configure CPU isolation using cpusets¶
Available since MOSK 22.2 TechPreview
The Linux kernel and cpuset provide a mechanism to run tasks by limiting the
resources defined by a cpuset. The tasks can be moved from one cpuset to
another to use the resources defined in other cpusets. The cset
Python tool
is a command-line interface to work with cpusets.
To configure CPU isolation using cpusets:
Configure core isolation:
Note
You can also automate this step during deployment by using the
postDeploy
script as described in Create MOSK host profiles.cat <<-"EOF" > /usr/bin/setup-cgroups.sh #!/bin/bash set -x UNSHIELDED_CPUS=${UNSHIELDED_CPUS:-"0-3"} SHIELD_CPUS=${SHIELD_CPUS:-"4-15"} SHIELD_MODE=${SHIELD_MODE:-"cpuset"} # One of isolcpu or cpuset DOCKER_CPUS=${DOCKER_CPUS:-$UNSHIELDED_CPUS} KUBERNETES_CPUS=${KUBERNETES_CPUS:-$UNSHIELDED_CPUS} CSET_CMD=${CSET_CMD:-"python2 /usr/bin/cset"} if [[ ${SHIELD_MODE} == "cpuset" ]]; then ${CSET_CMD} set -c ${UNSHIELDED_CPUS} -s system ${CSET_CMD} proc -m -f root -t system ${CSET_CMD} proc -k -f root -t system fi ${CSET_CMD} set --cpu=${DOCKER_CPUS} --set=docker ${CSET_CMD} set --cpu=${KUBERNETES_CPUS} --set=kubepods ${CSET_CMD} set --cpu=${DOCKER_CPUS} --set=com.docker.ucp EOF chmod +x /usr/bin/setup-cgroups.sh cat <<-"EOF" > /etc/systemd/system/shield-cpus.service [Unit] Description=Shield CPUs DefaultDependencies=no After=systemd-udev-settle.service Before=lvm2-activation-early.service Wants=systemd-udev-settle.service [Service] ExecStart=/usr/bin/setup-cgroups.sh RemainAfterExit=true Type=oneshot Restart=on-failure #Service should restart on failure RestartSec=5s #Restart each five seconds until success [Install] WantedBy=basic.target EOF systemctl enable shield-cpus reboot
As root user, verify that isolation has been applied:
cset set -l
Example of system response:
cset: Name CPUs-X MEMs-X Tasks Subs Path ------------ ---------- - ------- - ----- ---- ---------- root 0-15 y 0 y 165 4 / kubepods 0-3 n 0 n 0 2 /kubepods docker 0-3 n 0 n 0 0 /docker system 0-3 n 0 n 65 0 /system com.docker.ucp 0-3 n 0 n 0 0 /com.docker.ucp
Run the
cpustress
container:docker run -it --name cpustress --rm containerstack/cpustress --cpu 4 --timeout 30s --metrics-brief
Verify that isolated cores are not affected:
htop
Example of system response highlighting the load created on all available Docker cores: