Kernel neighbor table overflow on OpenStack controller nodes in large clusters

On MOSK clusters with more than 50 OpenStack compute nodes, default kernel neighbor table and socket buffer limits on OpenStack controller nodes can be insufficient for the volume of ARP entries and control-plane traffic the controllers handle. When the neighbor table overflows, host networking degrades and readiness or liveness probes on the controller nodes start timing out.

Symptoms:

You may observe the following symptoms on OpenStack controller nodes, often already during the initial cluster deployment:

  • Pods repeatedly restart because readiness or liveness probes time out. Typical examples include:

    • Ingress NGINX failing to reach 127.0.0.1:10246 with i/o timeout

    • Octavia readiness probe curl --fail http://localhost:9876 timing out

    • Calico felix-ready or felix-live probes timing out

  • The host kernel log contains the following messages:

    neighbour: arp_cache: neighbor table overflow!
net_ratelimit: <number> callbacks suppressed

Note

Localhost probe timeouts are a side effect of host networking congestion. Do not start by retuning Ingress, Octavia, or Calico. Inspect the kernel neighbor table and the messages above first.

To resolve the issue, increase the kernel neighbor table and socket buffer limits on OpenStack controller nodes as described in Tune kernel parameters on OpenStack controller nodes.