Troubleshoot node alerts

This section describes the investigation and troubleshooting steps for the node alerts.

ZombieProcessesDetected

Root cause

Zombie processes (state Z) are child processes that have exited but have not been reaped by their parent. A high number of zombies usually indicates one of the following:

  • A buggy or stuck parent process that does not call wait() on its children.

  • A parent process (for example, init, systemd, or a container entrypoint) that is not reaping due to overload or a bug.

  • A kernel or runtime issue on the node.

Risk

Zombie processes do not consume CPU or memory but consume PID slots. If the count keeps increasing, it may lead to PID exhaustion and prevent new processes from starting.

Investigation

  1. Identify the affected node from the alert labels.

  2. Log in to this node and open an interactive shell.

  3. List zombie processes and their parents. For example:

    echo "Zombies:" && ps -eo pid,ppid,stat,comm | awk '$3 ~ /Z/' && echo "Zombies count by parent:" && ps -eo ppid,stat | awk '$2 ~ /Z/ {print $1}' | sort | uniq -c && echo "Parents:" && ps -eo ppid,stat,comm | awk '$2 ~ /Z/ {print $1}' | xargs ps -o pid,ppid,stat,comm -p

  4. If the parent is PID 1 (init/systemd) or a well-known daemon, verify its logs and resource usage. Otherwise, follow the mitigation steps below.

Mitigation

  • A small number of zombies is often acceptable — they are reaped when the parent exits or the node reboots.

  • If the parent is a specific service or container, restart that workload (for example, restart the Pod or the systemd unit on the node) so the parent process exits and zombies are reaped.

  • If zombies keep growing or the parent cannot be restarted safely, plan a node drain and reboot to clear them and avoid PID exhaustion.