MKE backups no longer require pausing the reconciler and deleting MKE containers, and backing up a MKE manager does not disrupt the manager’s activities.
Because MKE stores the same data on all manager nodes, you only need to back up a single MKE manager node.
User resources, such as services, containers, and stacks are not affected by this operation and continue operating as expected.
Backup contents are stored in a .tar
file. Backups contain MKE
configuration metadata to re-create configurations such as
Administration Settings values such as LDAP and SAML, and RBAC
configurations (Collections, Grants, Roles, User, and more):
Data | Description | Bac ked up |
---|---|---|
Conf igurations | MKE configurations, including Mirantis Container Runtime license. Swarm, and client CAs | yes |
Access control | Permissions for teams to swarm resources, including collections, grants, and roles | yes |
Ce rtificates and keys | Certificates and public and private keys used for authentication and mutual TLS communication | yes |
Metrics data | Monitoring data gathered by MKE | yes |
Org anizations | Users, teams, and organizations | yes |
Volumes | All MKE named volumes including all MKE component certificates and data. | yes |
Overlay Networks | Swarm-mode overlay network definitions, including port information | no |
Configs, Secrets | Create a Swarm backup to backup these data | no |
Services | Stacks and services are stored in Swarm-mode or SCM/Config Management | no |
Note
Because Kubernetes stores the state of resources on etcd
, a
backup of etcd
is sufficient for stateless backups.
ucp-metrics-data
: holds the metrics server’s data.ucp-node-certs
: holds certs used to lock down MKE system
componentsMKE backups include all Kubernetes declarative objects (pods,
deployments, replicasets, configurations, and so on), including secrets.
These objects are stored in the ucp-kv etcd
database that is backed
up (and restored) as part of MKE backup/restore.
Note
You cannot back up Kubernetes volumes and node labels. Instead, upon restore, Kubernetes declarative objects are re-created. Containers are re-created and IP addresses are resolved.
For more information, see Backing up an etcd cluster.
To avoid directly managing backup files, you can specify a file name and
host directory on a secure and configured storage backend, such as NFS
or another networked file system. The file system location is the backup
folder on the manager node file system. This location must be writable
by the nobody
user, which is specified by changing the folder
ownership to nobody
. This operation requires administrator
permissions to the manager node, and must only be run once for a given
file system location.
sudo chown nobody:nogroup /path/to/folder
Important
Specify a different name for each backup file. Otherwise, the existing backup file with the same name is overwritten. Specify a location that is mounted on a fault-tolerant file system (such as NFS) rather than the node’s local disk. Otherwise, it is important to regularly move backups from the manager node’s local disk to ensure adequate space for ongoing backups.
There are several options for creating a MKE backup:
The backup process runs on one manager node.
The following example shows how to create a MKE manager node backup,
encrypt it by using a passphrase, decrypt it, verify its contents, and
store it locally on the node at /tmp/mybackup.tar
:
Run the
docker/ucp:3.2.5 backup
command on a single MKE manager and include the --file
and
--include-logs
options. This creates a tar archive with the
contents of all volumes used by MKE and
streams it to stdout
. Replace 3.2.5
with the
version you are currently running.
$ docker container run \
--rm \
--log-driver none \
--name ucp \
--volume /var/run/docker.sock:/var/run/docker.sock \
--volume /tmp:/backup \
docker/ucp:3.2.5 backup \
--file mybackup.tar \
--passphrase "secret12chars" \
--include-logs=false
Note
If you are running with Security-Enhanced Linux (SELinux) enabled,
which is typical for RHEL hosts, you must include
--security-opt label=disable
in the docker
command (replace
version
with the version you are currently running):
$ docker container run \
--rm \
--log-driver none \
--security-opt label=disable \
--name ucp \
--volume /var/run/docker.sock:/var/run/docker.sock \
docker/ucp:3.2.5 backup \
--passphrase "secret12chars" > /tmp/mybackup.tar
Note
To determine whether SELinux is enabled in the engine, view the
host’s /etc/docker/daemon.json
file, and search for the string
"selinux-enabled":"true"
.
To view backup progress and error reporting, view the contents of the
stderr streams of the running backup container during the backup.
Progress is updated for each backup step, for example, after validation,
after volumes are backed up, after etcd
is backed up, and after
rethinkDB
. Progress is not preserved after the backup has completed.
In a valid backup file, 27 or more files are displayed in the list and
the ./ucp-controller-server-certs/key.pem
file is present. Ensure
the backup is a valid tar file by listing its contents, as shown in the
following example:
$ gpg --decrypt /directory1/directory2/backup.tar | tar --list
If decryption is not needed, you can list the contents by removing the
--decrypt flag
, as shown in the following example:
$ tar --list -f /directory1/directory2/backup.tar
To create a MKE backup using the UI:
The UI also provides the following options: - Display the status of a running backup - Display backup history - View backup contents
The MKE API provides three endpoints for managing MKE backups. You must be a MKE administrator to access these API endpoints.
You can create a backup with the POST: /api/ucp/backup
endpoint.
This is a JSON endpoint with the following arguments:
field name | JSON data type* | description |
---|---|---|
passphrase | string | Encryption passphrase |
noPassphrase | bool | Set to true if not using a
passphrase |
fileName | string | Backup file name |
includeLogs | bool | Specifies whether to include a log file |
hostPath | string | File system loca tion |
The request returns one of the following HTTP status codes, and, if successful, a backup ID.
$ curl -sk -H 'Authorization: Bearer $AUTHTOKEN' https://$UCP_HOSTNAME/api/ucp/backup \
-X POST \
-H "Content-Type: application/json" \
--data '{"encrypted": true, "includeLogs": true, "fileName": "backup1.tar", "logFileName": "backup1.log", "hostPath": "/secure-location"}'
200 OK
where:
$AUTHTOKEN
is your authentication bearer token if using auth
token identification.$UCP_HOSTNAME
is your MKE hostname.You can view all existing backups with the GET: /api/ucp/backups
endpoint. This request does not expect a payload and returns a list of
backups, each as a JSON object following the schema found in the Backup
schema section.
The request returns one of the following HTTP status codes and, if successful, a list of existing backups:
curl -sk -H 'Authorization: Bearer $AUTHTOKEN' https://$UCP_HOSTNAME/api/ucp/backups
[
{
"id": "0d0525dd-948a-41b4-9f25-c6b4cd6d9fe4",
"encrypted": true,
"fileName": "backup2.tar",
"logFileName": "backup2.log",
"backupPath": "/secure-location",
"backupState": "SUCCESS",
"nodeLocation": "ucp-node-ubuntu-0",
"shortError": "",
"created_at": "2019-04-10T21:55:53.775Z",
"completed_at": "2019-04-10T21:56:01.184Z"
},
{
"id": "2cf210df-d641-44ca-bc21-bda757c08d18",
"encrypted": true,
"fileName": "backup1.tar",
"logFileName": "backup1.log",
"backupPath": "/secure-location",
"backupState": "IN_PROGRESS",
"nodeLocation": "ucp-node-ubuntu-0",
"shortError": "",
"created_at": "2019-04-10T01:23:59.404Z",
"completed_at": "0001-01-01T00:00:00Z"
}
]
You can retrieve details for a specific backup using the
GET: /api/ucp/backup/{backup_id}
endpoint, where {backup_id}
is
the ID of an existing backup. This request returns the backup, if it
exists, for the specified ID, as a JSON object following the schema
found in the Backup schema section.
The request returns one of the following HTTP status codes, and if successful, the backup for the specified ID:
{backup_id}
The following table describes the backup schema returned by the GET
and LIST
APIs:
field name | JSON data type* | description |
---|---|---|
id | string | Unique ID |
en crypted | boolean | Set to true if encrypted with a passphrase |
f ileName | string | Backup file name if backing up to a file, empty otherwise |
logF ileName | string | Backup log file name if saving backup logs, empty otherwise |
bac kupPath | string | Host path where backup resides |
back upState | string | Current state of the backup (IN_PROGRESS ,
SUCCESS , FAILED ) |
nodeL ocation | string | Node on which the backup was taken |
sho rtError | string | Short error. Empty unless backupState is
set to FAILED |
cre ated_at | string | Time of backup creation |
compl eted_at | string | Time of backup completion |