Create backups of Mirantis Kubernetes Engine

Caution

Remote backup storage and encryption can be configured on clusters that are already updated to 2.31.0 and 26.1.

This section describes how to create a backup manually or schedule automatic backups of Mirantis Kubernetes Engine (MKE) for management and MOSK clusters using the ClusterBackup and ClusterBackupSchedule custom resources provided by MOSK management API.

Important

Using remote storage and encrypting backup data is highly recommended for production environments.

Create an MKE backup manually

  1. Configure backup storage. Select one of the following options:

    • For rclone, create and configure the RCloneCredential object. For details, see RCloneCredential resource.

    • For NFS, configure it to allow mounting the share onto the cluster manager nodes.

      Note

      NFS versions 3 and 4 are supported by the Kubernetes in-tree storage driver. Mounting is performed using the kubelet.

    • Not recommended for production environments. Skip this step to store the backup data locally on one of the cluster manager nodes.

  2. Applies if remote storage is used. Create and configure the KeyEncryptionKey object to encrypt the backup data before uploading it to remote storage. For details, see KeyEncryptionKey resource.

    When the KeyEncryptionKey object is configured, the backup will be encrypted automatically after creation of the ClusterBackup object as described below.

  3. Create and configure the ClusterBackup object. For details, see ClusterBackup resource.

    Note

    Mirantis highly recommends that you use separate per-cluster folders in remote storage to easily identify the latest backup file for each cluster.

    Caution

    While the ClusterBackup object is in the Pending, InProgress, or Stuck status, a new backup cannot be started for the same cluster.

    To configure default settings for future remote backups during cluster updates, update the backup section in the Cluster object:

    spec:
  ...
  providerSpec:
    value:
      ...
      backup:
        keyEncryptionKey: my-key-encryption-key
        rclone:
          bucket: my-bucket
          credential: my-rclone-credential

  4. Apply the ClusterBackup object to the target cluster:

    kubectl apply -f <clusterbackup-object-name>.yaml

  5. Monitor the status of the ClusterBackup object. For details, see API Reference: ClusterBackup - status.

    Once the backup is completed, the backup file will be encrypted and uploaded to remote storage, if configured.

    The encrypted backup contains the actual backup file with the .encrypted suffix appended to the file name and the .dek file with the same name containing the data encryption key, which is encrypted by the key encryption key.

See also

Troubleshoot remote backup failures

If a remote backup fails before or after update, the execution of ClusterUpdatePlan stops meaning that the remote upload job has failed on the target cluster. The ClusterBackup object status contains a message referencing the job that failed, for example:

message: 'Backup upload job kaas/backup-5fn7jbtg751451t152uwspg3nbr168ke46rhjwkpjvqk-upload failed: Job has reached the specified backoff limit'

The issue may be caused by interrupted remote connectivity, incorrect RCloneCredential object values, or other issues. After fixing the issue, retrigger the backup upload by deleting the referenced job on the target cluster, which will be re-created automatically. For example:

kubectl delete job kaas/backup-5fn7jbtg751451t152uwspg3nbr168ke46rhjwkpjvqk-upload

Note

Mirantis highly recommends testing remote upload by creating the ClusterBackup object manually before trying to update the cluster with remote backup enabled.

Schedule MKE backups

You can schedule backups for MKE using the ClusterBackupSchedule resource. Once the ClusterBackupSchedule object is created, the corresponding ClusterBackup objects will be scheduled for creation at the specified interval.

Backup schedule is described in the standard cron schedule format. For example, set 0 0 * * * for daily backups at 00:00. The schedule is set to the time zone of the management cluster.

Caution

The minimum supported backup interval is one hour.

If during the scheduled backup time another ClusterBackup is running for the cluster, this object will be labeled by backup-controller as kaas.mirantis.com/clusterbackupschedule-reference: my-schedule and will be used instead of creating a new scheduled backup object.

Delete an MKE backup from remote storage

You can delete an MKE backup from remote storage using the corresponding ClusterBackup object by setting the delete field to true in the spec section. For details, see API Reference: ClusterBackup - spec.delete.

Deletion of the ClusterBackup object itself only triggers deletion of related Kubernetes objects created during backup. Therefore, always request deletion from the remote storage backend using the delete field before deleting the ClusterBackup object.

Deletion of the ClusterBackupSchedule object does not delete any ClusterBackup objects but only stops the creation of new ones.

If the target cluster is deleted, all ClusterBackup and ClusterBackupSchedule objects for this cluster are deleted automatically.