Configure CPU isolation for an instance

Note

Consider this section as part of Deploy an OpenStack cluster.

CPU isolation is a way to force the system scheduler to use only some logical CPU cores for processes. For compute hosts, you should typically isolate system processes and virtual guests on different cores. This section describes the two possible options on how to achieve this:

  • Through the isolcpus configuration parameter for Linux kernel Deprecated since MOSK 22.2

  • Through the cpusets mechanism in Linux kernel Available since MOSK 22.2, TechPreview

For details, see OpenStack official documentation: CPU topologies and Shielding Linux Resources.

Configure CPU isolation using isolcpus

Note

Starting from MOSK 22.2, isolcpus is deprecated.

Using the isolcpus parameter, specific CPUs are removed from the general kernel symmetrical multiprocessing (SMP) load balancing and scheduling. The only way to get tasks scheduled onto isolated CPUs is taskset. The list of isolcpus is configured statically at boot time. You can only change it by rebooting with a different value. In Linux kernel, the isolcpus parameter is deprecated in favor of cpusets.

To configure CPU isolation using isolcpus:

  1. Configure isolcpus in Linux kernel:

    GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=4-15"
    
  2. Apply the changes:

    update-grub
    
  3. Isolate cores from scheduling of Docker or Kubernetes workloads:

    cat <<-"EOF" > /usr/bin/setup-cgroups.sh
    #!/bin/bash
    
    set -x
    
    UNSHIELDED_CPUS=${UNSHIELDED_CPUS:-"0-3"}
    SHIELD_CPUS=${SHIELD_CPUS:-"4-15"}
    SHIELD_MODE=${SHIELD_MODE:-"isolcpu"} # One of isolcpu or cpuset
    
    DOCKER_CPUS=${DOCKER_CPUS:-$UNSHIELDED_CPUS}
    KUBERNETES_CPUS=${KUBERNETES_CPUS:-$UNSHIELDED_CPUS}
    CSET_CMD=${CSET_CMD:-"python2 /usr/bin/cset"}
    
    if [[ ${SHIELD_MODE} == "cpuset" ]]; then
        ${CSET_CMD} set -c ${UNSHIELDED_CPUS} -s system
        ${CSET_CMD} proc -m -f root -t system
        ${CSET_CMD} proc -k -f root -t system
    fi
    
    ${CSET_CMD} set --cpu=${DOCKER_CPUS} --set=docker
    ${CSET_CMD} set --cpu=${KUBERNETES_CPUS} --set=kubepods
    ${CSET_CMD} set --cpu=${DOCKER_CPUS} --set=com.docker.ucp
    
    EOF
    chmod +x /usr/bin/setup-cgroups.sh
    
    cat <<-"EOF" > /etc/systemd/system/shield-cpus.service
    [Unit]
    Descriptio=Shield CPUs
    DefaultDependencies=no
    After=systemd-udev-settle.service
    Before=lvm2-activation-early.service
    Wants=systemd-udev-settle.service
    [Service]
    ExecStart=/usr/bin/setup-cgroups.sh
    RemainAfterExit=true
    Type=oneshot
    [Install]
    WantedBy=basic.target
    EOF
    
    systemctl enable shield-cpus
    
  4. As root user, reboot the host:

    cat /sys/devices/system/cpu/isolated
    

    Example of system response:

    4-15
    
  5. As root user, verify that isolation is active:

    cset set -l
    

    Example of system response:

    cset:
         Name       CPUs-X      MEMs-X   Tasks Subs   Path
     ------------ ---------- - ------- - ----- ---- ----------
     root             0-15 y       0 y    1449    3  /
     kubepods         0-3 n        0 n       0    2  /kubepods
     docker           0-3 n        0 n       0    5  /docker
     com.docker.ucp   0-3 n        0 n       0    1  /com.docker.ucp
    

Configure CPU isolation using cpusets

Available since MOSK 22.2 TechPreview

The Linux kernel and cpuset provide a mechanism to run tasks by limiting the resources defined by a cpuset. The tasks can be moved from one cpuset to another to use the resources defined in other cpusets. The cset Python tool is a command-line interface to work with cpusets.

To configure CPU isolation using cpusets:

  1. Configure core isolation:

    Note

    You can also automate this step during deployment by using the postDeploy script as described in Create MOSK host profiles.

    cat <<-"EOF" > /usr/bin/setup-cgroups.sh
    #!/bin/bash
    
    set -x
    
    UNSHIELDED_CPUS=${UNSHIELDED_CPUS:-"0-3"}
    SHIELD_CPUS=${SHIELD_CPUS:-"4-15"}
    SHIELD_MODE=${SHIELD_MODE:-"cpuset"} # One of isolcpu or cpuset
    
    DOCKER_CPUS=${DOCKER_CPUS:-$UNSHIELDED_CPUS}
    KUBERNETES_CPUS=${KUBERNETES_CPUS:-$UNSHIELDED_CPUS}
    CSET_CMD=${CSET_CMD:-"python2 /usr/bin/cset"}
    
    if [[ ${SHIELD_MODE} == "cpuset" ]]; then
        ${CSET_CMD} set -c ${UNSHIELDED_CPUS} -s system
        ${CSET_CMD} proc -m -f root -t system
        ${CSET_CMD} proc -k -f root -t system
    fi
    
    ${CSET_CMD} set --cpu=${DOCKER_CPUS} --set=docker
    ${CSET_CMD} set --cpu=${KUBERNETES_CPUS} --set=kubepods
    ${CSET_CMD} set --cpu=${DOCKER_CPUS} --set=com.docker.ucp
    
    EOF
    chmod +x /usr/bin/setup-cgroups.sh
    
    cat <<-"EOF" > /etc/systemd/system/shield-cpus.service
    [Unit]
    Descriptio=Shield CPUs
    DefaultDependencies=no
    After=systemd-udev-settle.service
    Before=lvm2-activation-early.service
    Wants=systemd-udev-settle.service
    [Service]
    ExecStart=/usr/bin/setup-cgroups.sh
    RemainAfterExit=true
    Type=oneshot
    [Install]
    WantedBy=basic.target
    EOF
    
    systemctl enable shield-cpus
    
    reboot
    
  2. As root user, verify that isolation has been applied:

    cset set -l
    

    Example of system response:

    cset:
          Name       CPUs-X     MEMs-X    Tasks Subs   Path
      ------------ ---------- - ------- - ----- ---- ----------
      root             0-15 y       0 y     165    4  /
      kubepods         0-3 n        0 n       0    2  /kubepods
      docker           0-3 n        0 n       0    0  /docker
      system           0-3 n        0 n      65    0  /system
      com.docker.ucp   0-3 n        0 n       0    0  /com.docker.ucp
    
  3. Run the cpustress container:

    docker run -it --name cpustress --rm containerstack/cpustress --cpu 4 --timeout 30s --metrics-brief
    
  4. Verify that isolated cores are not affected:

    htop
    

    Example of system response highlighting the load created on all available Docker cores:

    ../../../../_images/cpu-isolation-htop.png