Frequent node disconnections with mcc-keepalived forcing new election

After cluster update, some nodes may remain in an unstable Ready state with mcc-keepalived constantly reelecting the leader and failing to acquire the VIP address, which produces forcing new election messages in logs.

To resolve the issue:

  1. Identify the leader node that owns the VIP:

    1. On any control plane node, run the following command:

      cat /etc/keepalived/keepalived.conf
      

      In the system response, capture the VIP used for the cluster.

    2. Using the VIP, identify the leader node:

      ip a| grep <VIP>
      

      If the VIP is not found, run the command on another control plane node until you find the leader.

  2. Connect to the non-leader control plane nodes and change the priority on these nodes in keepalived.conf:

    vi /etc/keepalived/keepalived.conf
    

    For example, change the priority on each node to 150 and 200 respectively:

    vrrp_instance VRRP1 {
        state MASTER
        garp_master_delay 15
        interface k8s-lcm
        virtual_router_id 154
        priority 100        # Change it on one node to 150 and on the other node to 200
        virtual_ipaddress {
            10.205.88.181
        }
    
  3. Restart the mcc-keepalived service on the control plane nodes where the priority was changed:

    systemctl restart mcc-keepalived
    
  4. In 10-15 minutes, verify the logs of the node identified in step 1:

    journalctl -u mcc-keepalived -f | grep election
    

    You should no longer see the forcing new election messages, and the flapping node status should be resolved.