Perform a graceful reboot of a cluster

Available since 2.23.0

You can perform a graceful reboot on a management or managed cluster. Use the below procedure to cordon, drain, and reboot the required cluster machines using a rolling reboot without workloads interruption. The procedure is also useful for a bulk reboot of machines, for example, on large clusters.

The reboot occurs in the order of cluster upgrade policy that you can change for managed clusters as described in Change the upgrade order of a machine or machine pool.


The cluster and machines must have the Ready status to perform a graceful reboot.

Perform a rolling reboot of a cluster using web UI

Available since 2.24.0 and 2.24.2 for MOSK 23.2

  1. Log in to the Container Cloud web UI with the m:kaas:namespace@operator or m:kaas:namespace@writer permissions.

  2. Switch to the required project using the Switch Project action icon located on top of the main left-side navigation panel.

  3. On the Clusters page, verify that the status of the required cluster is Ready. Otherwise, the Reboot machines option is disabled.

  4. Click the More action icon in the last column of the required cluster and select Reboot machines. Confirm the selection.


    While a graceful reboot is in progress, the Reboot machines option is disabled.

    To monitor the cluster readiness, hover over the status icon of a specific cluster in the Status column of the Clusters page.

    Once the orange blinking status icon becomes green and Ready, the cluster deployment or update is complete.

    You can monitor live deployment status of the following cluster components:




    For the OpenStack-based management clusters, the Bastion node IP address status that confirms the Bastion node creation


    Installation or upgrade status of all Helm releases


    Readiness of the node in a Kubernetes cluster, as reported by kubelet


    Readiness of all requested Kubernetes objects


    Equality of the requested nodes number in the cluster to the number of nodes having the Ready LCM status


    Readiness of the cluster OIDC configuration


    Health of all StackLight-related objects in a Kubernetes cluster


    Readiness of all nodes in a Docker Swarm cluster


    Readiness of the Kubernetes API load balancer


    Readiness of all machines in the underlying infrastructure (virtual or bare metal, depending on the provider type)

    Graceful Reboot

    Readiness of a cluster during a scheduled graceful reboot, available since Cluster releases 15.0.1 and 14.0.0.

    Infrastructure Status

    Available since Container Cloud 2.25.0 for bare metal and OpenStack providers. Readiness of the following cluster components:

    • Bare metal: the MetalLBConfig object along with MetalLB and DHCP subnets.

    • OpenStack: cluster network, routers, load balancers, and Bastion along with their ports and floating IPs.

    LCM Operation

    Available since Container Cloud 2.26.0 (Cluster releases 17.1.0 and 16.1.0). Health of all LCM operations on the cluster and its machines.

    LCM Agent

    Available since Container Cloud 2.27.0 (Cluster releases 17.2.0 and 16.2.0). Health of all LCM agents on cluster machines and the status of LCM agents update to the version from the current Cluster release.

    For the history of a cluster deployment or update, refer to Inspect the history of a cluster and machine deployment or update.


Machine configuration changes are forbidden during graceful reboot. Therefore, either wait until reboot is completed or cancel it using CLI, as described in the following section.

Perform a rolling reboot of a cluster using CLI

Available since 2.23.0

  1. Create a GracefulRebootRequest resource with a name that matches the name of the required cluster. For the resource fields description, see GracefulRebootRequest resource.

  2. In spec:machines, add the machine list or leave it empty to reboot all cluster machines.

    Wait until all specified machines are rebooted. You can monitor the reboot status of the cluster and machines using the Conditions:GracefulReboot fields of the corresponding Cluster and Machine objects.

    The GracefulRebootRequest object is automatically deleted once the reboot on all target machines completes.

    To monitor the live machine status:

    kubectl get machines <machineName> -o wide

    Example of system response:

    demo-0  true   Ready     kaas-node-c6aa8ad3  1             true


Machine configuration changes are forbidden during graceful reboot.

In emergency cases, for example, to migrate StackLight or Ceph services from a disabled machine that fails during graceful reboot and blocks the process, cancel the reboot by deleting the GracefulRebootRequest object:

kubectl -n <projectName> delete gracefulrebootrequest <gracefulRebootRequestName>

Once you migrate StackLight or Ceph services to another machine and disable it, re-create the GracefulRebootRequest object for the remaining machines that require reboot.


To reboot a single node, for example, for maintenance purposes, refer to Enable cluster and machine maintenance mode.