Automatic restore of the OpenSDN databases fails

If the automatic restore of the OpenSDN databases fails, the data is not fully imported. MOSK leaves dbRestoreMode enabled and does not start the OpenSDN configuration services on the databases that the restore did not finish.

In this case, you can observe the following:

  • status.health set to Failed in the tf-dbrestore object

  • DBRestoreJobFailed in status.eventList of the tf-dbrestore object and no DBRestoreFinished

  • spec.features.dbRestoreMode.enabled still set to true in the TFOperator custom resource

  • Missing tf-config and tf-config-db DaemonSets

Warning

After a failed restore, do not set dbRestoreMode to false manually. This starts the OpenSDN configuration services on partial or empty databases.

To resolve the issue:

  1. Inspect the logs of the tf-dbrestore-job job:

    kubectl -n tf logs job/tf-dbrestore-job

  2. Inspect the tf-dbrestore object:

    kubectl -n tf describe tfdbrestores.tf.mirantis.com tf-dbrestore

  3. Optional. Specify the name of another backup in the dbDumpName parameter of the TFOperator custom resource:

    spec:
  features:
    dbRestoreMode:
      enabled: true
      dbDumpName: db-dump-20220111-110138.json

  4. To retry the restore, delete the tf-dbrestore object while dbRestoreMode is still enabled:

    kubectl -n tf delete tfdbrestores.tf.mirantis.com tf-dbrestore

    The operator creates the tf-dbrestore object again and repeats the restore: it recreates the databases, verifies Cassandra and ZooKeeper, and creates a new restore job.

  5. Continue the restoration as described in Automatically restore the data, starting from step 4.