Tungsten Fabric known issues

This section lists the Tungsten Fabric known issues with workarounds for the Mirantis OpenStack for Kubernetes release 24.2.

Note

For the Tungsten Fabric limitations, refer to Tungsten Fabric known limitations.

[13755] TF pods switch to CrashLoopBackOff after a simultaneous reboot

Rebooting all Cassandra cluster TFConfig or TFAnalytics nodes, maintenance, or other circumstances that cause the Cassandra pods to start simultaneously may cause a broken Cassandra TFConfig and/or TFAnalytics cluster. In this case, Cassandra nodes do not join the ring and do not update the IPs of the neighbor nodes. As a result, the TF services cannot operate Cassandra cluster(s).

To verify that a Cassandra cluster is affected:

Run the nodetool status command specifying the config or analytics cluster and the replica number:

kubectl -n tf exec -it tf-cassandra-<config/analytics>-dc1-rack1-<replica number> -c cassandra -- nodetool status

Example of system response with outdated IP addresses:

Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address         Load       Tokens       Owns (effective)  Host ID                               Rack
DN  <outdated ip>   ?          256          64.9%             a58343d0-1e3f-4d54-bcdf-9b9b949ca873  r1
DN  <outdated ip>   ?          256          69.8%             67f1d07c-8b13-4482-a2f1-77fa34e90d48  r1
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address          Load       Tokens       Owns (effective)  Host ID                               Rack
UN  <actual ip>      3.84 GiB   256          65.2%             7324ebc4-577a-425f-b3de-96faac95a331  rack1

Workaround:

Manually delete the Cassandra pod from the failed config or analytics cluster to re-initiate the bootstrap process for one of the Cassandra nodes:

kubectl -n tf delete pod tf-cassandra-<config/analytics>-dc1-rack1-<replica_num>

[40032] tf-rabbitmq fails to start after rolling reboot

Occasionally, RabbitMQ instances in tf-rabbitmq pods fail to enable the tracking_records_in_ets during the initialization process.

To work around the problem, restart the affected pods manually.

[40900] Cassandra DB infinite table creation/changing state in Tungsten Fabric

During initial deployment of a Tungsten Fabric cluster, there is a known issue where the Cassandra database may enter an infinite table creation or changing state. This results in the Tungsten Fabric configuration pods failing to reach the Ready state.

The root cause of this issue is a schema mismatch within Cassandra.

To verify whether the cluster is affected:

The symptoms of this issue can be observed by verifying the Tungsten Fabric configuration pods:

kubectl describe pod <TF_CONFIG_POD_NAME> -n tf

The following events might be observed:

Events:
  Type     Reason     Age                  From     Message
  ----     ------     ----                 ----     -------
  Warning  Unhealthy  35m (x64 over 78m)   kubelet  Readiness probe failed: contrail-svc-monitor: initializing (Database:Cassandra[] connection down)
  Warning  BackOff    30m (x128 over 63m)  kubelet  Back-off restarting failed container svc-monitor in pod tf-config-mfkfc_tf(fc77e6b6-d7b9-4680-bffc-618796a754af)
  Warning  BackOff    25m (x42 over 44m)   kubelet  Back-off restarting failed container api in pod tf-config-mfkfc_tf(fc77e6b6-d7b9-4680-bffc-618796a754af)
  Normal   Started    20m (x18 over 80m)   kubelet  Started container svc-monitor
  Warning  Unhealthy  14m (x90 over 70m)   kubelet  (combined from similar events): Readiness probe errored: rpc error: code = Unknown desc = container not running (38fc8f52b45a0918363e5617c0d0181d72a01435a6c1ec4021301e2a0e75805e)

The events above indicate that the configuration services remain in the initializing state after deployment due to inability to connect to the database. As a result, liveness and readiness probes fail, and the pods continuously restart.

Additionally, each node of Cassandra configuration database logs similar errors:

INFO  [MigrationStage:1] 2024-05-29 23:13:23,089 MigrationCoordinator.java:531 - Sending schema pull request to /192.168.159.144 at 1717024403089 with timeout 10000
ERROR [InternalResponseStage:123] 2024-05-29 23:13:24,792 MigrationCoordinator.java:491 - Unable to merge schema from /192.168.159.144
org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found a1de4100-1e02-11ef-8797-b7bb8876c3bc; expected a22ef910-1e02-11ef-a135-a747d50b5bce)
 at org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaData.java:1000)
 at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:953)
 at org.apache.cassandra.config.Schema.updateTable(Schema.java:687)
 at org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1495)
 at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1451)
 at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1413)
 at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1390)
 at org.apache.cassandra.service.MigrationCoordinator.mergeSchemaFrom(MigrationCoordinator.java:449)
 at org.apache.cassandra.service.MigrationCoordinator$Callback.response(MigrationCoordinator.java:487)
 at org.apache.cassandra.service.MigrationCoordinator$Callback.response(MigrationCoordinator.java:475)
 at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53)
 at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:69)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
 at java.lang.Thread.run(Thread.java:750)

Workaround:

To resolve this issue temporarily, restart the affected Cassandra pod:

kubectl delete pod <TF_CASSANDRA_CONFIG_POD_NAME> -n tf

After the pod is restarted, monitor the status of other Tungsten Fabric pods. If they become Ready within two minutes, the issue is resolved. Otherwise, inspect the latest Cassandra logs in other pods and restart any other pods exhibiting the same pattern of errors:

kubectl delete pod <ANOTHER_TF_CASSANDRA_CONFIG_POD_NAME> -n tf

Repeat the process until all affected pods become Ready.