Apply etcd Defragmentation#

The underlying etcd database can become fragmented over time, as keys are created, updated, and deleted. As a result, the disk space from deleted entries is not immediately reclaimed, which causes the on-disk size of etcd to increase beyond the actual size of the data set.

Defragmentation rewrites the database to reclaim the wasted space, which reduces disk usage and improves I/O efficiency. As such, to reclaim space and maintain cluster performance, you should periodically perform this maintenance operation.

MKE 4k provides defragmentation utilities through the etcd maintenance service. Currently, only manual defragmentation calls are supported.

To defragment an etcd cluster

For the procedure that follows, http_port defaults to 18088.

SSH to any controller node:

ssh -i <path_to_ssh_key> user@<controller_node_ip>

Verify that the etcd maintenance service is running:
```
curl http://localhost:<http_port>/health
```
Example response:
```
{"status":"OK"}
```

Check cluster status to verify that a defragmentation operation is not currently running.

This endpoint can be used to monitor the state of etcd members, as well as to check whether a defragmentation operation is already running. If a defragmentation is in progress, no further defragmentations should be issued. The maintenance service will prevent multiple defrags from being issued.

curl http://localhost:<http_port>/cluster/status

Example response:

{
  "DefragInProgress": false,
  "members": {
    "172.31.0.149": {
      "DBSize": "89 MB",
      "DBSizeInUse": "31 MB",
      "alarms": null,
      "etcdVersion": "3.5.18",
      "isLeader": false,
      "memberID": 1141391576588418377
    },
    "172.31.0.175": {
      "DBSize": "89 MB",
      "DBSizeInUse": "31 MB",
      "alarms": null,
      "etcdVersion": "3.5.18",
      "isLeader": true,
      "memberID": -3067575474720874990
    },
    "172.31.0.191": {
      "DBSize": "89 MB",
      "DBSizeInUse": "31 MB",
      "alarms": null,
      "etcdVersion": "3.5.18",
      "isLeader": false,
      "memberID": 5368207591176096260
    }
  }
}

Issue a cluster defragmentation.

The example issues a timeout of 10 minute for each member defragmentation, and a pause of 1 minute between each member defragmentation, to let the freshly defragged member catch up. Mirantis recommends that you adjust these parameters based on the size of your etcd database and the amount of time that has elapsed since you last ran a defragmentation operation.
```
curl -X POST http://localhost:18088/cluster/defrag?timeoutSecond=600&pauseSeconds=60
```

Monitor the state of the defragmentation operation.

You can do use the etcd maintenance service logs to monitor the defragmentation, or you can check the status endpoint as follows:

curl http://localhost:18088/cluster/status

Example response:

{
  "DefragInProgress": true,
  "members": {
    "172.31.0.149": {
      "DBSize": "22 MB",
      "DBSizeInUse": "22 MB",
      "alarms": null,
      "etcdVersion": "3.5.18",
      "isLeader": false,
      "memberID": 1141391576588418377
    },
    "172.31.0.175": {
      "DBSize": "22 MB",
      "DBSizeInUse": "22 MB",
      "alarms": null,
      "etcdVersion": "3.5.18",
      "isLeader": true,
      "memberID": -3067575474720874990
    },
    "172.31.0.191": {
      "DBSize": "22 MB",
      "DBSizeInUse": "22 MB",
      "alarms": null,
      "etcdVersion": "3.5.18",
      "isLeader": false,
      "memberID": 5368207591176096260
    }
  }
}

Defragmentation Locking

To maintain etcd cluster availability, MKE 4k uses a lock file to prevent you from simultaneously triggering multiple defragmentations. The host service automatically removes this lock file at the end of each defragmentatsion operation.

If a node that is running a defragmentation operation goes offline or has some other unforeseeable issue, the removal of the lock file may not occur, which will prevent defragmentation operations going forward. In such an event, you can run the following command on the affected node to manually remove the lock file:

rm /var/lib/etcd-maintenance/defrag.lock

Be aware that you should only run this command in adverse situations as a last resort.