Tungsten Fabric known issues

This section lists the Tungsten Fabric (TF) known issues with workarounds for the Mirantis OpenStack for Kubernetes release 23.1. For TF limitations, see Tungsten Fabric known limitations.

[30738] ‘tf-vrouter-agent’ readiness probe failed (No Configuration for self)

Fixed in 23.3 Fixed in 23.2.1

Execution of the TF Heat Tempest test test_template_global_vrouter_config can result in lost vRouter configuration. This causes the tf-vrouter pod readiness probe to fail with the following error message:

"Readiness probe failed: vRouter is PRESENT contrail-vrouter-agent: initializing (No Configuration for self)"

As a result, vRouters may have an incomplete routing table making some services, such as metadata, become unavailable.

Workaround:

  1. Add the tf_heat_tempest_plugin tests with global configuration to the exclude list in the OpenStackDeployment custom resource:

    spec:
  tempest:
    tempest:
      values:
        conf:
          blacklist:
            - (?:tf_heat_tempest_plugin.tests.functional.test_global.*)

  2. If you ran test_template_global_vrouter_config and tf-vrouter-agent pods moved to the error state with the above error, re-create these pods through deletion:

    kubectl -n tf delete pod tf-vrouter-agent-*

[13755] TF pods switch to CrashLoopBackOff after a simultaneous reboot

Rebooting all Cassandra cluster TFConfig or TFAnalytics nodes, maintenance, or other circumstances that cause the Cassandra pods to start simultaneously may cause a broken Cassandra TFConfig and/or TFAnalytics cluster. In this case, Cassandra nodes do not join the ring and do not update the IPs of the neighbor nodes. As a result, the TF services cannot operate Cassandra cluster(s).

To verify that a Cassandra cluster is affected:

Run the nodetool status command specifying the config or analytics cluster and the replica number:

kubectl -n tf exec -it tf-cassandra-<config/analytics>-dc1-rack1-<replica number> -c cassandra -- nodetool status

Example of system response with outdated IP addresses:

Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address         Load       Tokens       Owns (effective)  Host ID                               Rack
DN  <outdated ip>   ?          256          64.9%             a58343d0-1e3f-4d54-bcdf-9b9b949ca873  r1
DN  <outdated ip>   ?          256          69.8%             67f1d07c-8b13-4482-a2f1-77fa34e90d48  r1
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address          Load       Tokens       Owns (effective)  Host ID                               Rack
UN  <actual ip>      3.84 GiB   256          65.2%             7324ebc4-577a-425f-b3de-96faac95a331  rack1

Workaround:

Manually delete the Cassandra pod from the failed config or analytics cluster to re-initiate the bootstrap process for one of the Cassandra nodes:

kubectl -n tf delete pod tf-cassandra-<config/analytics>-dc1-rack1-<replica_num>

[32723] Compiled vRouter kernel module does not refresh with the new kernel

Fixed in 23.2

The vRouter kernel module remains at /usr/src/vrouter-<TF-VROUTER-IMAGE-VERSION>, even if it was initially compiled for an older kernel version. This leads to the reuse of compiled artifacts without recompilation. Consequently, after upgrading to Mirantis OpenStack for Kubernetes 23.1, an outdated module gets loaded onto the new kernel. This mismatch results in a failure that triggers the CrashLoop state for the vRouter on the affected node.

Workaround:

  1. On the affected node, move the old vRouter kernel module to another directory. For example:

    mkdir /root/old_vrouter_kmods
mv /lib/modules/`uname -r`/updates/dkms/vrouter* /root/old_vrouter_kmods
mv /usr/src/vrouter-21.4.20230306000000 /root/old_vrouter_kmods

  2. On the cluster, remove the tf-vrouter-agent pod:

    kubectl delete -n tf pod tf-vrouter-agent-<POD_ID>