Tune kernel parameters on OpenStack controller nodes

On MOSK clusters with more than 50 OpenStack compute nodes, OpenStack controller nodes keep a large neighbor (ARP) cache and handle a high volume of control-plane traffic. Default kernel limits are often insufficient for this scale.

When the kernel neighbor table overflows, host networking degrades. Readiness and liveness probes start timing out, and Kubernetes restarts pods on the controller nodes. The failures often resemble application issues in Ingress NGINX, Octavia, Calico, and so on, while the root cause is on the host.

Mirantis recommends increasing the kernel parameters below on OpenStack controller nodes. This section explains how to apply them in one of the following ways:

During deployment planning, through the BareMetalHostProfile objects of affected OpenStack controller nodes.
After deployment, by using the sysctl host operating system configuration module.

Caution

The values in this section are recommended starting values for clusters with more than 50 OpenStack compute nodes. They are not a final configuration for every cloud. Depending on the number of nodes, network topology, and workload, you may need to further increase these parameters.

Caution

Do not apply these configuration recommendations to compute nodes. Configure only the OpenStack controller nodes if needed.

Symptoms of insufficient kernel limits

You may observe the following symptoms on OpenStack controller nodes, often already during the initial cluster deployment:

Pods repeatedly restart because readiness or liveness probes time out. Typical examples include:
- Ingress NGINX failing to reach 127.0.0.1:10246 with i/o timeout
- Octavia readiness probe curl --fail http://localhost:9876 timing out
- Calico felix-ready or felix-live probes timing out

The host kernel log contains the following messages:

neighbour: arp_cache: neighbor table overflow!
net_ratelimit: <number> callbacks suppressed

Note

Localhost probe timeouts are a side effect of host networking congestion. Do not start by retuning Ingress, Octavia, or Calico. Inspect the kernel neighbor table and the messages above first.

Recommended kernel parameters

Recommended `sysctl` parameters for OpenStack controller nodes
Parameter	Recommended value	Description
`net.ipv4.neigh.default.gc_thresh1`	`8192`	Minimum number of ARP neighbor entries to keep. The kernel does not garbage-collect the neighbor table while the number of entries is below this threshold.
`net.ipv4.neigh.default.gc_thresh2`	`65536`	Soft maximum of ARP neighbor entries. When the number of entries exceeds this value, the kernel garbage-collects the table more aggressively.
`net.ipv4.neigh.default.gc_thresh3`	`131072`	Hard maximum of ARP neighbor entries. When the table reaches this limit, the kernel rejects new neighbor entries and logs `neighbour: arp_cache: neighbor table overflow!`.
`net.core.rmem_default`	`4194304`	Default size of the receive buffer allocated for network sockets, in bytes (4 MiB).
`net.core.rmem_max`	`10485760`	Maximum size of the receive buffer that a socket can request, in bytes (10 MiB).
`net.core.wmem_default`	`4194304`	Default size of the send buffer allocated for network sockets, in bytes (4 MiB).
`net.core.wmem_max`	`10485760`	Maximum size of the send buffer that a socket can request, in bytes (10 MiB).

The gc_thresh* parameters raise the capacity of the IPv4 neighbor table so that a controller can keep entries for a large number of compute nodes and other cluster endpoints. The rmem_* and wmem_* parameters increase default and maximum socket buffer sizes for the amount of control-plane traffic on these nodes.

For the list of kernel options that you must not change, refer to MKE documentation: Set up kernel default protections.

Apply the parameters using a host profile

Use this procedure when you plan a new cluster or add or replace OpenStack controller nodes. The kernelParameters.sysctl values of a BareMetalHostProfile object are written to /etc/sysctl.d/999-baremetal.conf during bare-metal host provisioning.

Note

Changing kernelParameters.sysctl in a host profile does not reconfigure already provisioned hosts. For existing OpenStack controller nodes, use the sysctl module. Update the controller host profile as well, so that newly provisioned controller nodes receive the same settings.

To apply the recommended kernel parameters through a host profile:

Open the BareMetalHostProfile object that you use for OpenStack controller nodes as described in Create MOSK host profiles.

In the spec.kernelParameters.sysctl section, add the following parameters. Keep the existing options that the profile already defines.

spec:
  kernelParameters:
    sysctl:
      net.ipv4.neigh.default.gc_thresh1: "8192"
      net.ipv4.neigh.default.gc_thresh2: "65536"
      net.ipv4.neigh.default.gc_thresh3: "131072"
      net.core.rmem_default: "4194304"
      net.core.rmem_max: "10485760"
      net.core.wmem_default: "4194304"
      net.core.wmem_max: "10485760"

Apply the updated host profile:

kubectl --kubeconfig <pathToManagementClusterKubeconfig> -n <projectName> apply -f <pathToBareMetalHostProfileFile>

Apply the parameters using the sysctl module

TechPreview

Use this procedure on already deployed OpenStack controller nodes. The sysctl module applies kernel parameters through the host operating system configuration API without rebuilding the node.

To apply the recommended kernel parameters using the sysctl module:

Download kubeconfig of the management cluster as described in Connect to a MOSK cluster.

On the Machine objects of the OpenStack controller nodes, set a dedicated label that you will use in machineSelector. For example:

kubectl --kubeconfig <pathToManagementClusterKubeconfig> -n <projectName> \
  label machine <controllerMachineName> <labelName>=<labelValue>

Add the sysctl module configuration to an existing HostOSConfiguration object or create a new one. Use the sysctl module version that is available in the mcc-modules object of your cluster.

apiVersion: kaas.mirantis.com/v1alpha1
kind: HostOSConfiguration
metadata:
  name: os-controller-sysctl
  namespace: <projectName>
spec:
  machineSelector:
    matchLabels:
      <labelName>: "<labelValue>"
  configs:
  - module: sysctl
    moduleVersion: <sysctlModuleVersion>
    description: Recommended kernel parameters for OpenStack controller nodes
    values:
      filename: os-controller
      cleanup_before: true
      state: present
      options:
        net.ipv4.neigh.default.gc_thresh1: "8192"
        net.ipv4.neigh.default.gc_thresh2: "65536"
        net.ipv4.neigh.default.gc_thresh3: "131072"
        net.core.rmem_default: "4194304"
        net.core.rmem_max: "10485760"
        net.core.wmem_default: "4194304"
        net.core.wmem_max: "10485760"

Apply the HostOSConfiguration object:

kubectl --kubeconfig <pathToManagementClusterKubeconfig> -n <projectName> apply -f <pathToHostOSConfigurationFile>

Verify that the configuration is applied. The status field of the HostOSConfiguration object must contain no errors:
```
kubectl --kubeconfig <pathToManagementClusterKubeconfig> -n <projectName> \
  get hoc os-controller-sysctl -o yaml
```
For the status field description, see HostOSConfiguration status.