Skip to content

Maintenance operation scheduling#

You can schedule etcd maintenance tasks to run at a specific time.

Important

  • To minimize the impact on cluster operations, Mirantis recommends that you schedule maintenance operations to run during times of low cluster activity.
  • Maintenance operations must be scheduled at least 72 hours (3 days) apart. This minimum interval is enforced by validation and prevents excessive cluster disruption.
  • Cron scheduling relies on controller machine system time, thus you should make sure to schedule the cron for the appropriate time zone. For example, if controller machines are running in UTC, then the cron schedule should also be set in UTC.

Configuration#

Configure etcd maintenance scheduling in the mke4.yaml configuration file under spec.etcd.maintenanceService. Refer to the table below for configuration parameter details:

Parameter                          Description
cleanupEnabled Enables scheduled cleanup operations. Required for defragmentation.
defragEnabled Optional. Enables scheduled defragmentation. Requires that cleanup operations take place first.
cronExpression Cron expression that defines when maintenance operations are to be run.

Maintenance operation runs must be scheduled at least 72 hours apart.
minTTLToKeepSeconds Optional. Sets the minimum TTL, in seconds, for events to retain.
defragPauseSeconds Optional. Sets the pause duration between defragmentation operations on members.
defragTimeoutSeconds Optional. Sets the timeout duration for defragmentation operations.

Example configuration

apiVersion: mke.mirantis.com/v1alpha1
kind: MkeConfig
metadata:
  name: mke
  namespace: mke
spec:
  etcd:
    maintenanceService:
      cleanupEnabled: true
      defragEnabled: true
      cronExpression: "0 0 2 * * SUN"
      defragPauseSeconds: 60
      defragTimeoutSeconds: 600

Format the cron expression#

The etcd maintenance service supports two cron expression formats:

  • Full crontab specification
  • Descriptor
Type                &nbsp Format                                                     ;      Detail
Full crontab specification <seconds> <minutes> <hours> <day-of-month> <month> <day-of-week> Field Order:
  • Seconds (0-59)
  • Minutes (0-59)
  • Hours (0-23)
  • Day of month (1-31)
  • Month (1-12 or JAN-DEC)
  • Day of week (0-6 or SUN-SAT, where 0/7 = Sunday)

Examples:
  • 0 0 2 * * SUN (every Sunday at 2:00 AM; seven days between runs)
  • 0 0 0 1 * * (first day of every month at midnight; 30+ days between runs)
  • 0 0 0 * * MON (every Monday at midnight; seven days between runs)

    As some calendar-based expressions may fail to be at least 72 hours apart during specific end-of-month boundaries, you should take extra care when using such expressions.
Descriptor @ prefix

Simplified expressions for common intervals. Less precise and not recommended if defragmentation is enabled.		 Standard descriptors:
  • @yearly or @annually (once per year, January 1st at midnight)
  • @monthly (once per month, first day of the month at midnight)
  • @weekly (once per week, Sunday at midnight)
Standard descriptors that do not meet the 72-hour minimum interval requirement cannot be used for etcd maintenance scheduling, such as @daily.
Interval descriptors:
  • @every (In hours, must be at least 72)

Examples:
  • @weekly (every Sunday at midnight)
  • @monthly (first day of each month at midnight)
  • @every 72h (every three days)

Verification#

Following configuration, verify that the schedule is working:

  1. SSH into any controller node:

    ssh -i <path_to_ssh_key> user@<controller_node_ip>

  2. Check service status. The etcd maintenance service runs as a systemd service on each control plane node.

  3. Check service logs to determine when the maintenance operations have been run:

    sudo journalctl -u etcdmaintenance -f

  4. Use the HTTP API to monitor etcd status:

    curl http://localhost:18088/cluster/status