Skip to content

etcd maintenance operations auditing#

The etcd maintenance service audits every maintenance attempt, recording when it ran and whether it succeeded. Auditing cannot be disabled.

What the audit includes#

The etcd maintenance service writes one record per completed operation, capturing both successes and failures:

  • One record is written each time a cluster defragmentation completes, whether it succeeds or fails.
  • One record is written each time a Kubernetes event cleanup completes, whether it succeeds or fails. Dry-run cleanups are intentionally not audited.

Where the records are stored#

Audit records are written into the etcd cluster that is being maintained, as JSON values under append-only key prefixes:

  • Defragmentation: /mke4k/audit/defrag/
  • Cleanup: /mke4k/audit/cleanup/

Each record is stored under a unique key of the form <prefix>/<timestamp>-<nodeID>-<uuid>, where <timestamp> is an RFC3339 value and <uuid> is a randomly generated identifier. Thus, records accumulate as a durable history rather than overwriting one another, and they are sorted chronologically by key.

Record schema#

Each record is a JSON object with the following fields:

Field                   Type Description
timestamp string The time at which the operation completed, in RFC3339 format and UTC.
node_id string The node associated with the operation. For defragmentation, this is the leader node that coordinated the operation; for cleanup, it is the node that ran it.
outcome string Either success or failure.
error_detail string The error message. Present only when outcome is failure.

Example records:

// key: /mke4k/audit/defrag/2026-04-07T20:35:30Z-leader-1-<uuid>
{"timestamp":"2026-04-07T20:35:30Z","node_id":"leader-1","outcome":"success"}

// key: /mke4k/audit/cleanup/2026-04-07T20:36:30Z-node-a-<uuid>
{"timestamp":"2026-04-07T20:36:30Z","node_id":"node-a","outcome":"failure","error_detail":"cleanup failed"}

Read audit records#

No API or user interface exists for audit data. To read the records, use etcdctl to query etcd directly from a controller node:

  1. Connect to any controller node in the cluster with SSH:

    ssh -i <path_to_ssh_key> user@<controller_node_ip>

  2. Query the audit prefixes.

    Use etcdctl with the --prefix flag to retrieve all records under either prefix, supplying the endpoint and TLS credentials appropriate to your environment:

    etcdctl get /mke4k/audit/defrag/ --prefix
etcdctl get /mke4k/audit/cleanup/ --prefix

Because keys sort chronologically, the results are returned in the order in which the operations ran.

Limitations#

Be aware of the following constraints when you rely on audit data:

  • Retention is governed by etcd compaction, not by the maintenance service. The service applies no pruning, time-to-live (TTL), or rotation of its own. To preserve audit history, you must configure etcd with a sufficiently long auto-compaction retention — for example, through --auto-compact-retention. etcd cannot exclude specific key prefixes from compaction.
  • Because the records live inside the maintained etcd cluster, they are lost if etcd is reset or restored without the keys.
  • Auditing is best-effort. A failed audit write logs a warning only and never causes the maintenance operation itself to fail.
  • No audit record is written for dry-run cleanups.