- Home
- Configuration
- Backup
- Storage reliability for scheduled backups
Storage reliability for scheduled backups#
Schedule cluster backups covers how to
configure schedule. Herein, we cover how to choose and verify the storage
behind it so that a scheduled backup exists, is complete, and restores cleanly when
you need it.
Verify network storage stays available#
Warning
Use network-attached storage for schedule.path only with
replication: LeaderOnly. The warning in Schedule cluster
backups
explains the risk with AllControllers.
Configure the mount in /etc/fstab so that MKE 4 can detect when it is
missing. For how MKE 4 identifies the expected mount, reports on it, and
responds when it is missing, refer to Mount checks for scheduled
backups.
The mount check confirms the mount is present. It does not confirm that the server behind the mount is reachable or that reads and writes complete in a timely manner. A network share can remain mounted after its server becomes unreachable, and filesystem operations against it can block. For NFS, iSCSI, and other network-attached storage:
- Confirm the mounted source, filesystem type, available capacity, permissions, and write access from every controller, because any controller can become the leader.
- Repeat these checks after reboots, controller replacement, storage maintenance, and changes to the mount configuration.
- Monitor the server and network path independently, because the MKE 4 mount check does not detect a hung or unreachable server behind a mounted path.
Do not rely on LeaderOnly as a destination lock#
With replication: LeaderOnly, each controller checks whether it is the
current etcd leader when its own scheduled run starts. The controller does not
hold an exclusive lock on the backup destination for the duration of the run.
Verify behavior across a failover
A leadership change during a long-running backup can briefly overlap with
the new leader's own attempt. If LeaderOnly writes to shared storage,
test backup behavior through an actual controller failover rather than
assuming single-writer behavior.
Plan capacity beyond the temporary directory#
Temporary directory and disk space
covers space for the k0s snapshot staging directory. Confirm that
schedule.path also has room for a new archive in addition to every archive
that schedule.retain keeps. Monitor free space and inodes on that
filesystem and on the controller root filesystem.
With schedule.retain set to a fixed count, the archive count on a healthy
controller stabilizes rather than grows without bound. Investigate
unexpected growth, missing archives, or repeated creation failures.
Keep a copy outside the cluster#
MKE 4 writes scheduled backups to controller-managed storage. Copy the backups you rely on to storage outside the controller nodes and, where possible, outside the storage failure domain that the cluster depends on. A copy that shares infrastructure with the cluster does not protect against a failure of that infrastructure.
Monitor backup freshness#
Check the backup history and Schedule cluster
backups cover how
to read MkeBackup records and why MKE 4 can skip a run. Scope freshness
alerts to your replication setting:
- With
AllControllers, check freshness on each controller independently. - With
LeaderOnly, check for freshness across the cluster as a whole. A controller that is not the leader during a given window has no record for that window, which is expected.
After a significant configuration change, confirm that the next backup reflects the change. Check the backup history covers how to read a record's conditions for this purpose.
Validate recovery independently#
A Completed record confirms that MKE 4 produced an archive. Confirm the
rest yourself, at an interval that matches your recovery requirements:
- Inspect the destination storage directly for recent timestamps, plausible archive sizes, and retention history. These checks catch missing files and unexpected changes, but they do not confirm that an archive restores cleanly.
- Perform a test restore in an isolated environment, including a restore from your off-node copy, and verify the configuration and essential functions of the restored cluster. For restore requirements, refer to Restore an MKE 4 cluster.
Related topics#
- Schedule cluster backups
- Mount checks for scheduled backups
- Check the backup history
- Temporary directory and disk space
- Restore an MKE 4 cluster