Mirantis OpenStack for Kubernetes Operations Guide¶

Add a compute node¶

This section describes how to add a new compute node to your existing Mirantis OpenStack for Kubernetes deployment.

To add a compute node:

1. Add a bare metal host to the managed cluster with MOS as described in Mirantis Container Cloud Operations Guide: Add a bare metal host.

2. Create a Kubernetes machine in your cluster as described in Mitantis Container Cloud Operations Guide: Add a machine.

   When adding the machine, specify the node labels as required for an OpenStack compute node:

   OpenStack node roles¶

   Node role	Description	Kubernetes labels	Minimal count
   OpenStack control plane	Hosts the OpenStack control plane services such as database, messaging, API, schedulers, conductors, L3 and L2 agents.	openstack-control-plane=enabled openstack-gateway=enabled openvswitch=enabled	3
   OpenStack compute	Hosts the OpenStack compute services such as libvirt and L2 agents.	openstack-compute-node=enabled openvswitch=enabled (for a deployment with Open vSwitch as a back end for networking)	Varies

3. If required, configure the compute host to enable DPDK, huge pages, SR-IOV, and other advanced features in your MOS deployment. See MOS Deployment Guide: Advanced OpenStack configuration (optional) for details.

Delete a compute node¶

This section describes how to delete an OpenStack compute node from your MOS deployment.

To delete a compute node:

1. Migrate all workloads from the node. For more information, follow Nova official documentation: Migrate instances.

2. Ensure that there are no pods running on the node to delete by draining the node as instructed in the Kubernetes official documentation: Safely drain node.

3. Delete the node through the Mirantis Container Cloud web UI as described in Mirantis Container Cloud Operations Guide: Delete a machine.

Add a controller node¶

This section describes how to add a new control plane node to the existing MOS deployment.

To add an OpenStack controller node:

1. Add a bare metal host to the managed cluster with MOS as described in Mirantis Container Cloud Operations Guide: Add a bare metal host using CLI.

   When adding the bare metal host YAML file, specify the following OpenStack control plane node labels for the OpenStack control plane services such as database, messaging, API, schedulers, conductors, L3 and L2 agents:

   • openstack-control-plane=enabled

   • openstack-gateway=enabled

   • openvswitch=enabled

2. Create a Kubernetes machine in your cluster as described in Mirantis Container Cloud Operations Guide: Add a machine using CLI.

   When adding the machine, verify that OpenStack control plane node has the following labels:

   • openstack-control-plane=enabled

   • openstack-gateway=enabled

   • openvswitch=enabled

   Note

   Depending on the applications that were colocated on the failed controller node, you may need to specify some additional labels, for example, ceph_role_mgr=true and ceph_role_mon=true . To successfuly replace a failed mon and mgr node, refer to Mirantis Container Cloud Operations Guide: Manage Ceph.

3. Verify that the node is in the Ready state through the Kubernetes API:

   kubectl get node <NODE-NAME> -o wide | grep Ready
   

4. Verify that the node has all required labels described in the previous steps:

   kubectl get nodes --show-labels
   

5. Configure new Octavia health manager resources:

   1. Rerun the octavia-create-resources job:

      kubectl -n osh-system exec -t <OS-CONTROLLER-POD> -c osdpl osctl-job-rerun octavia-create-resources openstack
      

   2. Wait until the Octavia health manager pod on the newly added control plane node appears in the Running state:

      kubectl -n openstack get pods -o wide | grep <NODE_ID> | grep octavia-health-manager
      

      Note

      If the pod is in the crashloopbackoff state, remove it:

      kubectl -n openstack delete pod <OCTAVIA-HEALTH-MANAGER-POD-NAME>
      

   3. Verify that an OpenStack port for the node has been created and the node is in the Active state:

      kubectl -n openstack exec -t <KEYSTONE-CLIENT-POD-NAME> openstack port show octavia-health-manager-listen-port-<NODE-NAME>
      

Replace a failed controller node¶

This section describes how to replace a failed control plane node in your MOS deployment. The procedure applies to the control plane nodes that are, for example, permanently failed due to a hardware failure and appear in the NotReady state:

kubectl get nodes <CONTAINER-CLOUD-NODE-NAME>

Example of system response:

NAME                         STATUS       ROLES    AGE   VERSION
<CONTAINER-CLOUD-NODE-NAME>    NotReady   <none>   10d   v1.18.8-mirantis-1

To replace a failed controller node:

1. Remove the Kubernetes labels from the failed node by editing the .metadata.labels node object:

   kubectl edit node <CONTAINER-CLOUD-NODE-NAME>
   

2. Add the control plane node to your deployment as described in Add a controller node.

3. Identify all stateful applications present on the failed node:

   node=<CONTAINER-CLOUD-NODE-NAME>
   claims=$(kubectl -n openstack get pv -o jsonpath="{.items[?(@.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0] == '${node}')].spec.claimRef.name}")
   for i in $claims; do echo $i; done
   

   Example of system response:

   mysql-data-mariadb-server-2
   openstack-operator-bind-mounts-rfr-openstack-redis-1
   etcd-data-etcd-etcd-0
   

4. Reschedule stateful applications pods to healthy controller nodes as described in Reschedule stateful applications.

5. Remove the OpenStack port related to the Octavia health manager pod of the failed node:

   kubectl -n openstack exec -t <KEYSTONE-CLIENT-POD-NAME> openstack port delete octavia-health-manager-listen-port-<NODE-NAME>
   

Reschedule stateful applications¶

The rescheduling of stateful applications may be required when replacing a permanently failed node, decomissioning a node, migrating applications to nodes with a more suitable set of hardware, and in several other use cases.

MOS deployment profiles include the following stateful applications:

• OpenStack database (MariaDB)

• OpenStack coordination (etcd)

• OpenStack Time Series Database back end (Redis)

Each stateful application from the list above has a persistent volume claim (PVC) based on a local persistent volume per pod. Each of control plane nodes has a set of local volumes available. To migrate an application pod to another node, recreate a PVC with the persistent volume from the target node.

To reschedule pods to another control plane node:

1. Recreate a PVC on another control plane node:

   1. Select one of the persistent volumes available on the node:

      Caution

      A stateful application pod can only be migrated to the node that does not contain other pods of this application.

      NODE_NAME=<NODE-NAME>
      STORAGE_CLASS=$(kubectl -n openstack get osdpl <OSDPL_OBJECT_NAME> -o jsonpath='{.spec.local_volume_storage_class}')
      kubectl -n openstack get pv -o json | jq --arg NODE_NAME $NODE_NAME --arg STORAGE_CLASS $STORAGE_CLASS -r '.items[] | select(.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0] == $NODE_NAME and .spec.storageClassName == $STORAGE_CLASS and .status.phase == "Available") | .metadata.name'
      

   2. As the new PVC should contain the same parameters as the deleted one except for volumeName, save the old PVC configuration in YAML:

      kubectl -n <NAMESPACE> get pvc <PVC-NAME> -o yaml > <OLD-PVC>.yaml
      

      Note

      <NAMESPACE> is a Kubernetes namespace where the PVC is created. For Redis, specify openstack-redis, for other applications specify openstack.

   3. Delete the old PVC:

      kubectl -n <NAMESPACE> delete pvc <PVC-NAME>
      

      Note

      If a PVC has stuck in the terminating state, run kubectl -n openstack edit pvc <PVC-NAME> and remove the finalizers section from metadata of the PVC.

   4. Create the PVC with a new persistent volume:

      cat <<EOF | kubectl apply -f -
           apiVersion: v1
           kind: PersistentVolumeClaim
           metadata:
             name: <PVC-NAME>
             namespace: <NAMESPACE>
           spec:
             accessModes:
             - ReadWriteOnce
             resources:
               requests:
                 storage: <STORAGE-SIZE>
             storageClassName: <STORAGE-CLASS>
             volumeMode: Filesystem
             volumeName: <PV-NAME>
          EOF
      

      Caution

      <STORAGE-SIZE>, <STORAGE-CLASS>, and <NAMESPACE> should correspond to the storage, storageClassName, and namespace values from the <OLD-PVC>.yaml file with the old PVC configuration.

2. Reschedule the pods:

   • For MariaDB:

     1. Remove the pod:

        Note

        To remove a pod from a node in the NotReady state, add --grace-period=0 --force to the following command.

        kubectl -n openstack delete pod <STATEFULSET-NAME>-<NUMBER>
        

     2. Wait until the pod appears in the Ready state.

        When the rescheduling is finalized, the mariadb-server-2 pod joins back to the Galera cluster with a clean MySQL data directory and requests the Galera state transfer from the available nodes.

   • For etcd:

     1. Scale down the etcd StatefulSet to N-1 replicas, where N is the initial number of replicas in the etcd StatefulSet:

        kubectl -n openstack scale sts etcd-etcd --replicas=<N-1>
        

     2. Add the coordination section to the spec.services section of the OsDpl object:

        spec:
          services:
            coordination:
              etcd:
                values:
                  conf:
                    etcd:
                      ETCD_INITIAL_CLUSTER_STATE: existing
        

     3. Wait until all etcd pods appear in the Ready state.

     4. Scale the etcd StatefulSet to the initial number of replicas:

        kubectl -n openstack scale sts etcd-etcd --replicas=<NUMBER-OF-REPLICAS>
        

     5. Wait until all etcd pods appear in the Ready state.

     6. Remove the coordination section from the spec.services section of the OsDpl object.

     7. Wait until all etcd pods appear in the Ready state.

   • For Redis:

     1. Remove the pod:

        Note

        To remove a pod from a node in the NotReady state, add --grace-period=0 --force to the following command.

        kubectl -n openstack-redis delete pod <STATEFULSET-NAME>-<NUMBER>
        

     2. Wait until the pod appears in the Ready state.

Back up and restore a MariaDB Galera database¶

MOS uses a MariaDB Galera database cluster to store data generated by OpenStack components. Mirantis recommends backing up your databases daily to ensure the integrity of your data. Also, you should create an instant backup before upgrading your database or for testing purposes.

MOS uses the Mariabackup utility to back up MariaDB Galera cluster data. Mariabackup is launched on periodic basis by a Kubernetes cron job, which is part of the MOS installation and is in the suspended state by default. To start running the job, you need to explicitly enable it in the OpenStackDeployment object.

Also, MOS provides the means for the MariaDB Galera cluster restoration using Mariabackup. During restoration, the job restores all Mariadb Galera nodes.

Run Tempest tests¶

The OpenStack Integration Test Suite (Tempest), is a set of integration tests to be run against a live OpenStack cluster. This section instructs you on how to verify the workability of your OpenStack deployment using Tempest.

To verify an OpenStack deployment using Tempest:

1. Configure Tempest as required.

   To change the Tempest run parameters, use the following structure in the OsDpl CR:

   spec:
     services:
       tempest:
         tempest:
           values:
             conf:
               script: |
                 tempest run --config-file /etc/tempest/tempest.conf \
                 --concurrency 4 --blacklist-file /etc/tempest/test-blacklist --smoke
   

   The following example structure from the OsDpl CR will set image:build_timeout to 600 in the tempest.conf file:

   spec:
     services:
       tempest:
         tempest:
           values:
             conf:
               tempest:
                 image:
                   build_timeout: 600
   

2. Run Tempest. The OpenStack Tempest is deployed like other OpenStack services in a dedicated openstack-tempest Helm release by adding tempest to spec:features:services in the OSDPL resource.

   spec:
     features:
       services:
         - tempest
   

3. Wait until Tempest is ready. The Tempest tests are launched by the openstack-tempest-run-tests job. To keep track of the tests execution, run:

   kubectl -n openstack logs -l application=tempest,component=run-tests
   

4. Get the Tempest results. The Tempest results can be stored in a pvc-tempest PersistentVolumeClaim (PVC). To get them from a PVC, use:

   # Run pod and mount pvc to it
   cat <<EOF | kubectl apply -f -
   apiVersion: v1
   kind: Pod
   metadata:
     name: tempest-test-results-pod
     namespace: openstack
   spec:
     nodeSelector:
       openstack-control-plane: enabled
     volumes:
       - name: tempest-pvc-storage
         persistentVolumeClaim:
           claimName: pvc-tempest
     containers:
       - name: tempest-pvc-container
         image: ubuntu
         command: ['sh', '-c', 'sleep infinity']
         volumeMounts:
           - mountPath: "/var/lib/tempest/data"
             name: tempest-pvc-storage
   EOF
   

5. If required, copy the results locally:

   kubectl -n openstack cp tempest-test-results-pod:/var/lib/tempest/data/report_file.xml .
   

6. Remove the Tempest test results pod:

   kubectl -n openstack delete pod tempest-test-results-pod
   

7. To rerun Tempest:

   1. Remove Tempest from the list of enabled services.

   2. Wait until Tempest jobs are removed.

   3. Add Tempest back to the list of the enabled services.

Update OpenStack¶

The update of the OpenStack components is performed during the MOS managed cluster release update through the Container Cloud web UI as described in Mirantis Container Cloud Operations Guide: Update a managed cluster.

Calculate a maintenance window duration¶

This section provides the background information on the approximate time spent on operations for pods of different purposes, possible data plane impact during these operations, and the possibility of a parallel pods update. Such data helps the cloud administrators to correctly estimate maintenance windows and impacts on the workloads for your OpenStack deployment.

Remove an OpenStack cluster¶

This section instructs you on how to remove an OpenStack cluster, deployed on top of Kubernetes, by deleting the openstackdeployments.lcm.mirantis.com (OsDpl) CR.

To remove an OpenStack cluster:

1. Verify that the OsDpl object is present:

   kubectl get osdpl -n openstack
   

2. Delete the OsDpl object:

   kubectl delete osdpl osh-dev -n openstack
   

   The deletion may take a certain amount of time.

3. Verify that all pods and jobs have been deleted and no objects are present in the command output:

   kubectl get pods,jobs -n openstack
   

4. Delete the Redis failover object:

   kubectl get redisfailover -n openstack-redis
   kubectl delete redisfailover openstack-redis -n openstack-redis
   

5. Delete the Redis operator:

   kubectl delete helmbundle redis-operator -n osh-system
   

6. Delete Persistent Volume Claims (PVCs) using the following snippet. Deletion of PVCs causes data deletion on Persistent Volumes. The volumes themselves will become available for further operations.

   Caution

   Before deleting PVCs, save valuable data in a safe place.

   #!/bin/bash
   PVCS=$(kubectl get pvc  --all-namespaces | egrep "redis|etcd|mariadb" | awk '{print $1" "$2" "$4}'| column -t | awk 'NR>1')
   echo  "$PVCS" | while read line; do
   PVC_NAMESPACE=$(echo "$line" | awk '{print $1}')
   PVC_NAME=$(echo "$line" | awk '{print $2}')
   echo "Deleting PVC ${PVC_NAME}"
   kubectl delete pvc ${PVC_NAME} -n ${PVC_NAMESPACE}
   done
   

   Note

   Deletion of PVCs may get stuck if a resource that uses the PVC is still running. Once the resource is deleted, the PVC deletion process will proceed.

7. Delete the MariaDB state ConfigMap:

   kubectl delete configmap openstack-mariadb-mariadb-state -n openstack
   

8. Delete secrets using the following snippet:

   #!/bin/bash
   SECRETS=$(kubectl get secret  -n openstack | awk '{print $1}'| column -t | awk 'NR>1')
   echo  "$SECRETS" | while read line; do
   echo "Deleting Secret ${line}"
   kubectl delete secret ${line} -n openstack
   done
   

9. Verify that OpenStack ConfigMaps and secrets have been deleted:

   kubectl get configmaps,secrets -n openstack
   

Configure Ceph RGW TLS¶

Once you enable Ceph RGW as described in Mirantis Container Cloud: Enable Ceph RGW Object Storage, you can configure the Transport Layer Security (TLS) protocol for a Ceph RGW public endpoint using the following options:

• Using MOS TLS, if it is enabled and exposes its certificates and domain for Ceph. In this case, Ceph RGW will automatically create an ingress rule with MOS certificates and domain to access the Ceph RGW public endpoint. Therefore, you only need to reach the Ceph RGW public and internal endpoints and set the CA certificates for a trusted TLS connection.

• Using custom ingress specified in the KaaSCephCluster CR. In this case, Ceph RGW public endpoint will use the public domain specified using the ingress parameters.

To configure Ceph RGW TLS:

1. Verify whether MOS TLS is enabled. The spec.features.ssl.public_endpoints section should be specified in the OpenStackDeployment CR.

2. To generate an SSL certificate for internal usage, verify that the gateway securePort parameter is specified in the KaasCephCluster CR. For details, see Mirantis Container Cloud: Enable Ceph RGW Object Storage.

3. Select from the following options:

   • If MOS TLS is enabled, obtain the MOS CA certificate for a trusted connection:

     kubectl -n openstack-ceph-shared get secret openstack-rgw-creds -o jsonpath="{.data.ca_cert}" | base64 -d
     

   • Configure Ceph RGW TLS using a custom ingress:

     1. Open the KaasCephCluster CR for editing.

     2. Specify the ingress parameters:

        • publicDomain - domain name to use for the external service. For example, <rgw-name>.<publicDomain>.

        • cacert - Certificate Authority (CA) certificate, used for the ingress rule TLS support.

        • tlsCert - TLS certificate, used for the ingress rule TLS support.

        • tlsKey - TLS private key, used for the ingress rule TLS support.

        For example:

        rgw:
          ingress:
            domain: public.domain.name
            cacert: |
              -----BEGIN CERTIFICATE-----
              ...
              -----END CERTIFICATE-----
            tlsCert: |
              -----BEGIN CERTIFICATE-----
              ...
              -----END CERTIFICATE-----
            tlsKey: |
              -----BEGIN RSA PRIVATE KEY-----
              ...
              -----END RSA PRIVATE KEY-----
        

4. To access internal and public Ceph RGW endpoints:

   1. Obtain the Ceph RGW public endpoint:

      kubectl -n rook-ceph get ingress
      

   2. To use the Ceph RGW internal endpoint with TLS, configure trusted connection for the required CA certificate:

      kubectl -n rook-ceph get secret rgw-ssl-local-certificate -o jsonpath="{.data.cacert}" | base64 -d
      

   3. Obtain the internal endpoint name for Ceph RGW:

      kubectl -n rook-ceph get svc -l app=rook-ceph-rgw
      

      The internal endpoint for Ceph RGW has the https://<internal-svc-name>.rook-ceph.svc:<rgw-secure-port>/ format, where <rgw-secure-port> is spec.rgw.gateway.securePort specified in the KaaSCephCluster CR.

View Grafana dashboards¶

Using the Grafana web UI, you can view the visual representation of the metric graphs based on the time series databases.

This section describes only the OpenStack-related Grafana dashboards. For other dashboards, including the system, Kubernetes, Ceph, and StackLight dashboards, see Mirantis Container Cloud Operations Guide: View Grafana dashboards.

To view the Grafana dashboards:

1. Log in to the Grafana web UI as described in Mirantis Container Cloud Operations Guide: Access StackLight web UIs.

2. From the drop-down list, select the required dashboard to inspect the status and statistics of the corresponding service deployed in Mirantis OpenStack on Kubernetes:

   Dashboard	Description
   OpenStack - Overview	Provides general information on OpenStack services resources consumption, API errors, deployed OpenStack compute nodes and block storage usage.
   KPI - Downtime	Provides uptime statistics for OpenStack compute instances, including graphs on created VMs status and availability.
   KPI - Provisioning	Provides provisioning statistics for OpenStack compute instances, including graphs on VM creation results by day.
   Cinder	Provides graphs on the OpenStack Block Storage service health, HTTP API availability, pool capacity and utilization, number of created volumes and snapshots.
   Glance	Provides graphs on the OpenStack Image service health, HTTP API availability, number of created images and snapshots.
   Gnocchi	Provides panels and graphs on the Gnocchi health and HTTP API availability.
   Heat	Provides graphs on the OpenStack Orchestration service health, HTTP API availability and usage.
   Ironic	Provides graphs on the OpenStack Bare Metal Provisioning service health, HTTP API availability, provisioned nodes by state and installed ironic-conductor back-end drivers.
   Keystone	Provides graphs on the OpenStack Identity service health, HTTP API availability, number of tenants and users by state.
   Neutron	Provides graphs on the OpenStack networking service health, HTTP API availability, agents status and usage of Neutron L2 and L3 resources.
   NGINX Ingress controller	Monitors the number of requests, response times and statuses, as well as the number of Ingress SSL certificates including expiration time and resources usage.
   Nova - Availability Zones	Provides detailed graphs on the OpenStack availability zones and hypervisor usage.
   Nova - Hypervisor Overview	Provides a set of single-stat panels presenting resources usage by host.
   Nova - Instances	Provides graphs on libvirt Prometheus exporter health and resources usage. Monitors the number of running instances and tasks and allows sorting the metrics by top instances.
   Nova - Overview	Provides graphs on the OpenStack compute services (nova-scheduler, nova-conductor, and nova-compute) health, as well as HTTP API availability.
   Nova - Tenants	Provides graphs on CPU, RAM, disk throughput, IOPS, and space usage and allocation and allows sorting the metrics by top tenants.
   Nova - Users	Provides graphs on CPU, RAM, disk throughput, IOPS, and space usage and allocation and allows sorting the metrics by top users.
   Nova - Utilization	Provides detailed graphs on Nova hypervisor resources capacity and consumption.
   Memcached	Memcached Prometheus exporter dashboard. Monitors Kubernetes Memcached pods and displays memory usage, hit rate, evicts and reclaims rate, items in cache, network statistics, and commands rate.
   MySQL	MySQL Prometheus exporter dashboard. Monitors Kubernetes MySQL pods, resources usage and provides details on current connections and database performance.
   RabbitMQ	RabbitMQ Prometheus exporter dashboard. Monitors Kubernetes RabbitMQ pods, resources usage and provides details on cluster utilization and performance.
   Cassandra Available since MOS 21.1	Provides graphs on Cassandra clusters’ health, ongoing operations, and resource consumption.
   Kafka Clusters Available since MOS 21.1	Provides graphs on Kafka clusters’ and broker health, as well as broker and topic usage.
   Redis Clusters Available since MOS 21.1	Provides graphs on Redis clusters’ and pods’ health, connections, command calls, and resource consumption.
   Tungsten Fabric Controller Available since MOS 21.1	Provides graphs on the overall Tungsten Fabric controller cluster processes and usage.
   Tungsten Fabric vRouter Available since MOS 21.1	Provides graphs on the overall Tungsten Fabric vRouter cluster processes and usage.
   ZooKeeper Clusters Available since MOS 21.1	Provides graphs on ZooKeeper clusters’ quorum health and resource consumption.

View Kibana dashboards¶

Using the Kibana web UI, you can view the visual representation of your OpenStack deployment notifications.

This section describes only the Openstack-related Kibana dashboards. For other dashboards, including the logs and Kubernetes events dashboards, see Mirantis Container Cloud Operations Guide: View Kibana dashboards.

To view the Kibana dashboards:

1. Log in to the Kibana web UI as described in Mirantis Container Cloud Operations Guide: Access StackLight web UIs.

2. Click the required dashboard to inspect the visualizations or perform a search:

   Dashboard	Description
   Notifications	Provides visualizations on the number of notifications over time per source and severity, host, and breakdowns. Includes search.
   Audit	Provides search on audit notifications.

StackLight alerts¶

This section provides an overview of the available predefined OpenStack-related StackLight alerts. To view the alerts, use the Prometheus web UI. To view the firing alerts, use Alertmanager or Alerta web UI.

For other alerts, including the node, Kubernetes, Ceph, and StackLight alerts, see Mirantis Container Cloud Operations Guide: Available StackLight alerts.

Configure StackLight¶

This section describes how to configure StackLight in your Mirantis OpenStack on Kubernetes deployment and includes the description of OpenStack-related StackLight parameters and their verification. For other available configuration keys and their configuration verification, see Mirantis Container Cloud Operations Guide: Manage StackLight.

credentialsConfig:
  username: "stacklight"
  password: "stacklight"
  host: "rabbitmq.openstack.svc"
  queue: "notifications"
  vhost: "openstack"

credentialsDiscovery:
  enabled: true
  namespace: openstack
  secretName: os-rabbitmq-user-credentials

credentialsConfig:
  identityEndpoint: "" # "http://keystone-api.openstack.svc:5000/v3"
  domain: "" # "default"
  password: "" # "workshop"
  project: "" # "admin"
  region: "" # "RegionOne"
  username: "" # "admin"

credentialsDiscovery:
  enabled: true
  namespace: openstack
  secretName: keystone-keystone-admin

insecure:
  internal: true
  external: false

Update Tungsten Fabric¶

The Tungsten Fabric cluster update is performed during the MOS managed cluster release update through the Container Cloud web UI as described in Mirantis Container Cloud Operations Guide: Update a managed cluster.

Replace a failed TF controller node¶

If one of the Tungsten Fabric (TF) controller nodes has failed, follow this procedure to replace it with a new node.

To replace a TF controller node:

1. Delete the failed TF controller node from the Kubernetes cluster:

   kubectl delete node <FAILED-TF-CONTROLLER-NODE-NAME>
   

   Note

   Once the failed node has been removed from the cluster, all pods that hanged in the Terminating state should be removed.

2. Assign the TF labels for the new control plane node as per the table below using the following command:

   kubectl label node <NODE-NAME> <LABEL-KEY=LABEL-VALUE> ...
   

   Tungsten Fabric (TF) node roles¶

   Node role	Description	Kubernetes labels	Minimal count
   TF control plane	Hosts the TF control plane services such as database, messaging, api, svc, config.	tfconfig=enabled tfcontrol=enabled tfwebui=enabled tfconfigdb=enabled	3
   TF analytics	Hosts the TF analytics services.	tfanalytics=enabled tfanalyticsdb=enabled	3
   TF vRouter	Hosts the TF vRouter module and vRouter agent.	tfvrouter=enabled	Varies
   TF vRouter DPDK Technical Preview	Hosts the TF vRouter agent in DPDK mode.	tfvrouter-dpdk=enabled	Varies

   Note

   TF supports only Kubernetes OpenStack workloads. Therefore, you should label OpenStack compute nodes with the tfvrouter=enabled label.

   Note

   Do not specify the openstack-gateway=enabled and openvswitch=enabled labels for the MOS deployments with TF as a networking back end for OpenStack.

3. Once you label the new Kubernetes node, new pods start scheduling on the node. Though, pods that use Persistent Volume Claims are stuck in the Pending state as their volume claims stay bounded to the local volumes from the deleted node. To resolve the issue:

   1. Delete the PersistentVolumeClaim (PVC) bounded to the local volume from the failed node:

      kubectl -n tf delete pvc <PVC-BOUNDED-TO-NON-EXISTING-VOLUME>
      

      Note

      Clustered services that use PVC, such as Cassandra, Kafka, and ZooKeeper, start the replication process when new pods move to the Ready state.

   2. Delete the pod that is using the removed PVC:

      kubectl -n tf delete pod <POD-NAME>
      

4. Verify that the pods have successfully started on the replaced controller node and stay in the Ready state.

5. Delete terminated nodes from the TF configuration through the TF web UI:

   1. Log in to the TF web UI.

   2. Navigate to Configure > BGP Routers.

   3. Delete all terminated control nodes.

      Note

      You can manage nodes of other types from Configure > Nodes.

Verify the status of Tungsten Fabric services¶

This section describes how to verify the status of the Tungsten Fabric services using the TF Operator tf-status tool that resides in tf-tool-status containers among the services. Using this tool, you can also verify the status of the Cassandra, ZooKeeper, Kafka, Redis, and RabbitMQ third-party services supported by TF.

A tf-tool-status container provides access to the contrail-status Python script through HTTP. To verify the TF services status, use the following options:

• Run contrail-status.py directly from the tf-tool-status container console.

• Send an HTTP request to the Kubernetes node IP, using the tf-tool-status container port, which is 8888 by default. The tf-status pod is deployed as a DaemonSet and does not expose any Kubernetes service. For third-party services, the tool uses the tf-tool-status-party pod.

• Send an HTTP request to tf-tool-status-aggregator, a Kubernetes microservice that aggregates statuses from all hosts that handle at least one TF service or a third-party service.

  Caution

  For third-party services, due to a limitation, tf-tool-status-aggregator may return an error. To obtain the status of a service pod, use direct access or send an HTTP request to the tf-tool-status container.

• Verify the status of TF services

• Verify the status of third-party services

To verify the status of the TF services:

Select from the following options:

• Run contrail-status.py directly from the console of the tf-tool-status container:

  kubectl -n tf exec <TF-TOOL-STATUS-POD> -- python /root/contrail-status.py
  

  For the debug mode, use the --debug flag:

  kubectl -n tf exec <TF-TOOL-STATUS-POD> -- python /root/contrail-status.py --debug
  

• Send an HTTP request to the tf-tool-status container:

  1. Obtain the list of nodes that handle tf-tool-status containers:

     kubectl -n tf get pod --selector app=tf-tool-status -o wide
     

  2. For the required host, obtain its port provided by the container:

     kubectl -n tf get pod <TF-TOOL-STATUS-POD> -o custom-columns=NAME:.metadata.name,HOST:.spec.nodeName,HOST-IP:.status.hostIP,PORT:.spec.containers[0].ports[0].hostPort
     

     Example of system response:

     NAME                  HOST        HOST-IP      PORT
     tf-tool-status-xxxx   host_name   10.10.0.12   8888
     

  3. Using the obtained IP and port, request tf-tool-status from any node in the environment. For example:

     curl 10.10.0.12:8888
     
     # or
     
     curl 10.10.0.12:8888/json   # for output in JSON format
     
     # or
     
     curl 10.10.0.12:8888/debug  # for the debug mode
     

     This process may take a certain amount of time depending on the services on the host.

• Send an HTTP request to the tf-tool-status-aggregator Kubernetes service:

  1. Obtain the CLUSTER-IP address of the Kubernetes aggregator service:

     kubectl -n tf get svc --selector app=tf-tool-status-aggregator -o wide
     

     Example of system response:

     NAME                        TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
     tf-tool-status-aggregator   ClusterIP   10.96.145.46   <none>        80/TCP    46h
     

     The service from the example above is provided on port 80.

  2. Obtain the status of the service. Substitute <CLUSTER-IP> with the obtained tf-tool-status-aggregator service CLUSTER-IP.

     Note

     For help with tf-tool-status-aggregator, use curl <CLUSTER-IP>.

     • To obtain the list of tf-status pods and basic information about them:

       curl <CLUSTER-IP>/pod-list
       

       Example of system response:

       Name: tf-tool-status-2h4pq
       IP: 10.10.0.27
       UUID: a2feb3d1-62d5-11ea-b302-02420a0a010b
       HostIP: 10.10.0.27
       Host: ps-ws-w2myzajhovir-1-xskgrqlroqob-server-hqrfdu5kxhcc
       

     • To obtain an aggregated status of all TF services:

       # shows aggregated statuses sorted by host/pod
       curl <CLUSTER-IP>/status
       
       # output in JSON format by host/pod
       curl <CLUSTER-IP>/status/json
       
       # the same as /status
       curl <CLUSTER-IP>/status/node
       
       # shows aggregated statuses sorted by services, such as config-api, webui
       curl <CLUSTER-IP>/status/group