Verify the status of Tungsten Fabric services

Available since MOS Ussuri Update

This section describes how to verify the status of the Tungsten Fabric services using the TF Operator tf-status tool that resides in tf-tool-status containers among the services. Using this tool, you can also verify the status of the Cassandra, ZooKeeper, Kafka, Redis, and RabbitMQ third-party services supported by TF.

A tf-tool-status container provides access to the contrail-status Python script through HTTP. To verify the TF services status, use the following options:

  • Run contrail-status.py directly from the tf-tool-status container console.

  • Send an HTTP request to the Kubernetes node IP, using the tf-tool-status container port, which is 8888 by default. The tf-status pod is deployed as a DaemonSet and does not expose any Kubernetes service. For third-party services, the tool uses the tf-tool-status-party pod.

  • Send an HTTP request to tf-tool-status-aggregator, a Kubernetes microservice that aggregates statuses from all hosts that handle at least one TF service or a third-party service.

    Caution

    For third-party services, due to a limitation, tf-tool-status-aggregator may return an error. To obtain the status of a service pod, use direct access or send an HTTP request to the tf-tool-status container.

  • Verify the status of TF services

  • Verify the status of third-party services


To verify the status of the TF services:

Select from the following options:

  • Run contrail-status.py directly from the console of the tf-tool-status container:

    kubectl -n tf exec <TF-TOOL-STATUS-POD> -- python /root/contrail-status.py
    

    For the debug mode, use the --debug flag:

    kubectl -n tf exec <TF-TOOL-STATUS-POD> -- python /root/contrail-status.py --debug
    
  • Send an HTTP request to the tf-tool-status container:

    1. Obtain the list of nodes that handle tf-tool-status containers:

      kubectl -n tf get pod --selector app=tf-tool-status -o wide
      
    2. For the required host, obtain its port provided by the container:

      kubectl -n tf get pod <TF-TOOL-STATUS-POD> -o custom-columns=NAME:.metadata.name,HOST:.spec.nodeName,HOST-IP:.status.hostIP,PORT:.spec.containers[0].ports[0].hostPort
      

      Example of system response:

      NAME                  HOST        HOST-IP      PORT
      tf-tool-status-xxxx   host_name   10.10.0.12   8888
      
    3. Using the obtained IP and port, request tf-tool-status from any node in the environment. For example:

      curl 10.10.0.12:8888
      
      # or
      
      curl 10.10.0.12:8888/json   # for output in JSON format
      
      # or
      
      curl 10.10.0.12:8888/debug  # for the debug mode
      

      This process may take a certain amount of time depending on the services on the host.

  • Send an HTTP request to the tf-tool-status-aggregator Kubernetes service:

    1. Obtain the CLUSTER-IP address of the Kubernetes aggregator service:

      kubectl -n tf get svc --selector app=tf-tool-status-aggregator -o wide
      

      Example of system response:

      NAME                        TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
      tf-tool-status-aggregator   ClusterIP   10.96.145.46   <none>        80/TCP    46h
      

      The service from the example above is provided on port 80.

    2. Obtain the status of the service. Substitute <CLUSTER-IP> with the obtained tf-tool-status-aggregator service CLUSTER-IP.

      Note

      For help with tf-tool-status-aggregator, use curl <CLUSTER-IP>.

      • To obtain the list of tf-status pods and basic information about them:

        curl <CLUSTER-IP>/pod-list
        

        Example of system response:

        Name: tf-tool-status-2h4pq
        IP: 10.10.0.27
        UUID: a2feb3d1-62d5-11ea-b302-02420a0a010b
        HostIP: 10.10.0.27
        Host: ps-ws-w2myzajhovir-1-xskgrqlroqob-server-hqrfdu5kxhcc
        
      • To obtain an aggregated status of all TF services:

        # shows aggregated statuses sorted by host/pod
        curl <CLUSTER-IP>/status
        
        # output in JSON format by host/pod
        curl <CLUSTER-IP>/status/json
        
        # the same as /status
        curl <CLUSTER-IP>/status/node
        
        # shows aggregated statuses sorted by services, such as config-api, webui
        curl <CLUSTER-IP>/status/group
        

To verify the status of third-party services:

  1. Obtain the IP address and port of the tf-tool-status containers for third-party services:

    kubectl -n tf get pod -l tungstenfabric=status-party -o custom-columns=NAME:.metadata.name,IP:.status.podIP,PORT:.spec.containers[0].ports[0].containerPort
    

    Example of system response:

    NAME                                    IP              PORT
    tf-tool-status-party-7dfdb9dbb6-5pw8q   192.168.62.79   8080
    
  2. Request the third-party service status. For example:

    curl 192.168.62.79:8080