Inspect the history of a cluster and machine deployment or update

Available since 2.22.0

Using the ClusterDeploymentStatus, ClusterUpgradeStatus, MachineDeploymentStatus, and MachineUpgradeStatus objects, you can inspect historical data of cluster and machine deployment or update stages, their time stamps, statuses, and failure messages, if any.

Caution

The order of cluster and machine update stages may not always be sorted by a time stamp but have an approximate logical order due to several components running simultaneously.

View the history using the web UI

  1. Log in to the Container Cloud web UI with the m:kaas:namespace@operator or m:kaas:namespace@writer permissions.

  2. Switch to the required project using the Switch Project action icon on top of the main left-side navigation panel.

  3. In the Clusters tab, click the More action icon in the last column of the required cluster area and select History to display details of the ClusterDeploymentStatus or ClusterUpgradeStatus object, if any.

  4. In the window that opens, click the required object to display the object stages, their time stamps, and statuses.

    Object names match the initial and/or target Cluster release versions and MKE versions of the cluster at a specific date and time. For example, 11.6.0+3.5.5 (initial version) or 11.5.0+3.5.5 -> 11.6.0+3.5.5.

    If any stage fails, hover over the Failure status field to display the failure message.

  5. Optional. Inspect the deployment and update status of the cluster machines:

    1. In the Clusters tab, click the required cluster name. The cluster page with Machines list opens.

    2. Click More action icon in the last column of the required machine area and select History.

View the history using the CLI

  1. Log in to a local machine where your management cluster kubeconfig is located and where kubectl is installed.

  2. Select from the following options:

    • Inspect the cluster or machine deployment history using the ClusterDeploymentStatus or MachineDeploymentStatus object:

      ./bin/kubectl --kubeconfig <pathToClusterKubeconfig> get clusterdeploymentstatus
      
      ./bin/kubectl --kubeconfig <pathToClusterKubeconfig> get machinedeploymentstatus
      
      Example extract from a ClusterDeploymentStatus object
      kind: ClusterDeploymentStatus
      metadata:
        annotations:
          lcm.mirantis.com/new-history: "true"
        creationTimestamp: "2022-12-13T15:25:49Z"
        name: test-managed
        namespace: default
        ownerReferences:
        - apiVersion: cluster.k8s.io/v1alpha1
          kind: Cluster
          name: test-managed
      release: 11.6.0+3.5.5
      stages:
      - message: ""
        name: Network prepared
        status: Success
        success: true
        timestamp: "2022-12-13T15:27:19Z"
      - message: ""
        name: Load balancers created
        status: Success
        success: true
        timestamp: "2022-12-13T15:27:56Z"
      - message: ""
        name: IAM objects created
        status: Success
        success: true
        timestamp: "2022-12-13T15:27:21Z"
      - message: ""
        name: Kubernetes API started
        status: Success
        success: true
        timestamp: "2022-12-13T15:57:05Z"
      - message: ""
        name: Helm-controller deployed
        status: Success
        success: true
        timestamp: "2022-12-13T15:57:13Z"
      - message: ""
        name: HelmBundle created
        status: Success
        success: true
        timestamp: "2022-12-13T15:57:15Z"
      - message: ""
        name: Certificates configured
        status: Success
        success: true
        timestamp: "2022-12-13T15:58:29Z"
      - message: ""
        name: All machines of the cluster are ready
        status: Success
        success: true
        timestamp: "2022-12-13T16:04:49Z"
      - message: ""
        name: OIDC configured
        status: Success
        success: true
        timestamp: "2022-12-13T16:04:07Z"
      - message: ""
        name: Cluster is ready
        status: Success
        success: true
        timestamp: "2022-12-13T16:14:04Z"
      
      Example extract from a MachineDeploymentStatus object
      apiVersion: kaas.mirantis.com/v1alpha1
      kind: MachineDeploymentStatus
      metadata:
        creationTimestamp: "2022-12-16T09:57:07Z"
        name: test-managed-2-master-cwtf8
        namespace: default
        ownerReferences:
        - apiVersion: cluster.k8s.io/v1alpha1
          kind: Machine
          name: test-managed-2-master-cwtf8
      release: 11.6.0+3.5.5
      stages:
      - message: ""
        name: Provider instance ready
        status: Success
        success: true
        timestamp: "2022-12-16T09:59:28Z"
      - message: ""
        name: Prepare phase done
        status: Success
        success: true
        timestamp: "2022-12-16T10:12:05Z"
      - message: ""
        name: Deploy phase done
        status: Success
        success: true
        timestamp: "2022-12-16T10:16:36Z"
      - message: ""
        name: Reconfigure phase done
        status: Success
        success: true
        timestamp: "2022-12-16T10:16:40Z"
      
    • Inspect the cluster or machine update history using the ClusterUpgradeStatus and MachineUpgradeStatus objects:

      ./bin/kubectl --kubeconfig <pathToClusterKubeconfig> get clusterupgradestatus
      
      ./bin/kubectl --kubeconfig <pathToClusterKubeconfig> get machineupgradestatus
      
      Example extract from a ClusterUpgradeStatus object
      apiVersion: kaas.mirantis.com/v1alpha1
      fromRelease: 11.5.0+3.5.5
      kind: ClusterUpgradeStatus
      metadata:
        annotations:
          lcm.mirantis.com/new-history: "true"
        creationTimestamp: "2022-12-16T09:02:20Z"
        name: test-managed-11.6.0+3.5.5
        namespace: default
        ownerReferences:
        - apiVersion: cluster.k8s.io/v1alpha1
          kind: Cluster
          name: test-managed
      stages:
      - message: ""
        name: Requested cluster maintenance
        status: Success
        success: true
        timestamp: "2022-12-16T09:02:02Z"
      - message: ""
        name: Cluster workload locks are inactive
        status: Success
        success: true
        timestamp: "2022-12-16T09:02:02Z"
      - message: ""
        name: Worker nodes MKE upgraded
        status: Success
        success: true
        timestamp: "2022-12-16T09:47:48Z"
      - message: ""
        name: Control plane nodes MKE upgraded
        status: Success
        success: true
        timestamp: "2022-12-16T09:37:33Z"
      - message: ""
        name: Cluster maintenance request removed
        status: Success
        success: true
        timestamp: "2022-12-16T09:48:16Z"
      toRelease: 11.6.0+3.5.5
      
      Example extract from a MachineUpgradeStatus object for a control plane machine
      apiVersion: kaas.mirantis.com/v1alpha1
      fromRelease: 11.5.0+3.5.5
      kind: MachineUpgradeStatus
      metadata:
        annotations:
          lcm.mirantis.com/new-history: "true"
        creationTimestamp: "2022-12-16T09:02:24Z"
        name: test-managed-master-j2vmj-11.6.0+3.5.5
        namespace: default
        ownerReferences:
        - apiVersion: cluster.k8s.io/v1alpha1
          kind: Machine
          name: test-managed-master-j2vmj
      stages:
      - message: ""
        name: Agent upgraded
        status: Success
        success: true
        timestamp: "2022-12-16T09:02:22Z"
      - message: ""
        name: Machine Upgraded
        status: Success
        success: true
        timestamp: "2022-12-16T09:37:14Z"
      toRelease: 11.6.0+3.5.5
      
      Example extract from a MachineUpgradeStatus object for a worker machine
      apiVersion: kaas.mirantis.com/v1alpha1
      fromRelease: 11.5.0+3.5.5
      kind: MachineUpgradeStatus
      metadata:
        annotations:
          lcm.mirantis.com/new-history: "true"
        creationTimestamp: "2022-12-16T09:02:58Z"
        name: test-managed-node-ggzn2-11.6.0+3.5.5
        namespace: default
        ownerReferences:
        - apiVersion: cluster.k8s.io/v1alpha1
          kind: Machine
          name: test-managed-node-ggzn2
      stages:
      - message: ""
        name: Agent upgraded
        status: Success
        success: true
        timestamp: "2022-12-16T09:02:56Z"
      - message: ""
        name: Previous machines are deployed
        status: Success
        success: true
        timestamp: "2022-12-16T09:42:20Z"
      - message: ""
        name: Node maintenance requested
        status: Success
        success: true
        timestamp: "2022-12-16T09:42:13Z"
      - message: ""
        name: Node workload locks inactive
        status: Success
        success: true
        timestamp: "2022-12-16T09:42:14Z"
      - message: ""
        name: Kubernetes drained
        status: Success
        success: true
        timestamp: "2022-12-16T09:42:14Z"
      - message: ""
        name: Deploy phase done
        status: Success
        success: true
        timestamp: "2022-12-16T09:43:33Z"
      - message: ""
        name: Reconfigure phase done
        status: Success
        success: true
        timestamp: "2022-12-16T09:43:36Z"
      - message: ""
        name: Machine Upgraded
        status: Success
        success: true
        timestamp: "2022-12-16T09:48:15Z"
      - message: ""
        name: Kubernetes uncordoned
        status: Success
        success: true
        timestamp: "2022-12-16T09:47:47Z"
      - message: ""
        name: Node maintenance request removed
        status: Success
        success: true
        timestamp: "2022-12-16T09:47:48Z"
      toRelease: 11.6.0+3.5.5
      

    Object names match the initial and/or target Cluster release versions and MKE versions of the cluster. For example, 11.5.0+3.5.5 (initial version) or 11.5.0+3.5.5 -> 11.6.0+3.5.5. Each object displays the update stages, their time stamps, and statuses. If any stage fails, the success field contains a failure message.