Skip to content

Storage reliability for scheduled backups#

Schedule cluster backups covers how to configure schedule. Herein, we cover how to choose and verify the storage behind it so that a scheduled backup exists, is complete, and restores cleanly when you need it.

Verify network storage stays available#

Warning

Use network-attached storage for schedule.path only with replication: LeaderOnly. The warning in Schedule cluster backups explains the risk with AllControllers.

Configure the mount in /etc/fstab so that MKE 4 can detect when it is missing. For how MKE 4 identifies the expected mount, reports on it, and responds when it is missing, refer to Mount checks for scheduled backups.

The mount check confirms the mount is present. It does not confirm that the server behind the mount is reachable or that reads and writes complete in a timely manner. A network share can remain mounted after its server becomes unreachable, and filesystem operations against it can block. For NFS, iSCSI, and other network-attached storage:

  • Confirm the mounted source, filesystem type, available capacity, permissions, and write access from every controller, because any controller can become the leader.
  • Repeat these checks after reboots, controller replacement, storage maintenance, and changes to the mount configuration.
  • Monitor the server and network path independently, because the MKE 4 mount check does not detect a hung or unreachable server behind a mounted path.

Do not rely on LeaderOnly as a destination lock#

With replication: LeaderOnly, each controller checks whether it is the current etcd leader when its own scheduled run starts. The controller does not hold an exclusive lock on the backup destination for the duration of the run.

Verify behavior across a failover

A leadership change during a long-running backup can briefly overlap with the new leader's own attempt. If LeaderOnly writes to shared storage, test backup behavior through an actual controller failover rather than assuming single-writer behavior.

Plan capacity beyond the temporary directory#

Temporary directory and disk space covers space for the k0s snapshot staging directory. Confirm that schedule.path also has room for a new archive in addition to every archive that schedule.retain keeps. Monitor free space and inodes on that filesystem and on the controller root filesystem.

With schedule.retain set to a fixed count, the archive count on a healthy controller stabilizes rather than grows without bound. Investigate unexpected growth, missing archives, or repeated creation failures.

Keep a copy outside the cluster#

MKE 4 writes scheduled backups to controller-managed storage. Copy the backups you rely on to storage outside the controller nodes and, where possible, outside the storage failure domain that the cluster depends on. A copy that shares infrastructure with the cluster does not protect against a failure of that infrastructure.

Monitor backup freshness#

Check the backup history and Schedule cluster backups cover how to read MkeBackup records and why MKE 4 can skip a run. Scope freshness alerts to your replication setting:

  • With AllControllers, check freshness on each controller independently.
  • With LeaderOnly, check for freshness across the cluster as a whole. A controller that is not the leader during a given window has no record for that window, which is expected.

After a significant configuration change, confirm that the next backup reflects the change. Check the backup history covers how to read a record's conditions for this purpose.

Validate recovery independently#

A Completed record confirms that MKE 4 produced an archive. Confirm the rest yourself, at an interval that matches your recovery requirements:

  • Inspect the destination storage directly for recent timestamps, plausible archive sizes, and retention history. These checks catch missing files and unexpected changes, but they do not confirm that an archive restores cleanly.
  • Perform a test restore in an isolated environment, including a restore from your off-node copy, and verify the configuration and essential functions of the restored cluster. For restore requirements, refer to Restore an MKE 4 cluster.