Automatic restore of the Tungsten Fabric databases fails

If the automatic restore of the Tungsten Fabric databases fails, the data is not fully imported. MOSK leaves dbRestoreMode enabled and does not start the Tungsten Fabric configuration services on the databases that the restore did not finish.

In this case, you can observe the following:

status.health set to Failed in the tf-dbrestore object
DBRestoreJobFailed in status.eventList of the tf-dbrestore object and no DBRestoreFinished
spec.features.dbRestoreMode.enabled still set to true in the TFOperator custom resource
Missing tf-config and tf-config-db DaemonSets

Warning

After a failed restore, do not set dbRestoreMode to false manually. This starts the Tungsten Fabric configuration services on partial or empty databases.

To resolve the issue:

Inspect the logs of the tf-dbrestore-job job:

kubectl -n tf logs job/tf-dbrestore-job

Inspect the tf-dbrestore object:

kubectl -n tf describe tfdbrestores.tf.mirantis.com tf-dbrestore

Optional. Specify the name of another backup in the dbDumpName parameter of the TFOperator custom resource:

spec:
  features:
    dbRestoreMode:
      enabled: true
      dbDumpName: db-dump-20220111-110138.json

To retry the restore, delete the tf-dbrestore object while dbRestoreMode is still enabled:
```
kubectl -n tf delete tfdbrestores.tf.mirantis.com tf-dbrestore
```
The operator creates the tf-dbrestore object again and repeats the restore: it recreates the databases, verifies Cassandra and ZooKeeper, and creates a new restore job.
Continue the restoration as described in Automatically restore the data, starting from step 4.