Offline installation#

The installation and upgrade procedures for MKE 4 reflect those of the online scenario. While the online installation typically uses registry.mirantis.com as the primary OCI registry for MKE 4 materials, with the offline scenario you instead specify a private registry from which to pull the MKE 4 images and charts.

Dependencies#

An OCI-based private registry that is accessible from all cluster nodes:
- MKE 4 images and charts may either be served anonymously or from a registry that requires authentication. To pull from an authenticated registry, refer to Configure authenticated registry below.
- The registry must use HTTPS.
- The registry must support multi-level nesting. For example, registry.com/level-one/level-two/level-three/image-name:latest. Some registries only allow one level of nesting, such as registry.com/level-one/image:latest, so verify that your registry supports deeper nesting for image names.
skopeo 1.17.0 or later.

Preparation#

Verify that your skopeo installation is version 1.17.0 or later.
```
skopeo --version
```
If the version of skopeo you are running is earlier than 1.17.0, you must install a supported version. You can obtain the latest version of the utility from the skopeo opensuse.org page.

Download the MKE 4 offline bundle from the command line as follows:

MKE_VERSION="4.2.1"
DOWNLOAD_URL="https://packages.mirantis.com/caas/mke_bundle_v${MKE_VERSION}_amd64.tar.gz"
BUNDLE_NAME="mke_bundle_v${MKE_VERSION}_amd64.tar.gz"
curl -L "$DOWNLOAD_URL" -o "$BUNDLE_NAME"

Verify the checksum of the bundle to confirm the download:

curl -L https://github.com/MirantisContainers/mke-release/releases/download/v${MKE_VERSION}/mke_bundle_v${MKE_VERSION}_amd64_checksum.txt -o "$BUNDLE_NAME.sha256" sha256sum -c "$BUNDLE_NAME.sha256"

Transfer the bundle file to a machine that can access your private registry.

On the machine with registry access, set the environment variables:

MKE_VERSION="4.2.1"
export REGISTRY_ADDRESS='<registry_address>'
export REGISTRY_PROJECT_PATH='<registry-path>'
export REGISTRY_USERNAME='<username>'
export REGISTRY_PASSWORD='<password>'
export BUNDLE_NAME="mke_bundle_v${MKE_VERSION}_amd64.tar.gz"

Environment variable	Description
REGISTRY_ADDRESS	Registry hostname (required) and port (optional). The value must not end with a slash '/'. Example: `private-registry.example.com:8080`
REGISTRY_PROJECT_PATH	Path to the registry project that will store all MKE 4 artifacts. The registry address and path should comprise the full registry path. The value must not end with a slash '/'. Example: `REGISTRY_ADDRESS + '/' + REGISTRY_PROJECT_PATH == 'private-registry.example.com:8080/mke`
REGISTRY_USERNAME	Username for the account that is allowed to push.
REGISTRY_PASSWORD	Password for the account that is allowed to push.
BUNDLE_NAME	The name of previously downloaded bundle file, which must be located in the same directory in which you run the preparation steps.

Upload the MKE 4 images and helm charts to your private registry:

# Login to the registry
skopeo login "$REGISTRY_ADDRESS" -u "$REGISTRY_USERNAME" -p "$REGISTRY_PASSWORD"

# Extract the bundle
tar -xzf "$BUNDLE_NAME" -C ./

# Iterate over bundle artifacts and upload each one using skopeo
for archive in $(find ./bundle -print | grep ".tar"); do
  # Form the image name from the archive name
  img=$(basename "$archive" | sed 's~\.tar~~' | tr '&' '/' | tr '@' ':');

  echo "Uploading $img";
  # Copy artifact from local oci archive to the registry
  skopeo copy -q --retry-times 3 --multi-arch all "oci-archive:$archive" "docker://$REGISTRY_ADDRESS/$REGISTRY_PROJECT_PATH/$img";
done;

Note

If you will run Windows nodes in your MKE 4 cluster, repeat steps 2-6 with the Windows bundle, indicating BUNDLE_NAME as follows:

BUNDLE_NAME="mke_windows_bundle_v${MKE_VERSION}_amd64.tar.gz"

Installation#

Info

For information on performing an upgrade to an existing MKE 3 installation in an air gap environment, refer to Offline upgrade.

Refer to the Create a Cluster procedure for detail on how to create an mke4.yaml configuration file.

Add the following additional settings to the mke4.yaml configuration file:

Setting	Description
`.spec.registries.imageRegistry.URL`	Sets your registry address with a project path that contains your MKE 4 images. For example, `private-registry.example.com:8080/mke`. The setting must not end with a slash `/`. The port is optional. `.spec.registries.imageRegistry.URL` can only be configured once during the first running of the `mkectl apply` command.
`.spec.registries.chartRegistry.URL`	Sets your registry address with a project path that contains your MKE 4 helm charts in OCI format. For example, `oci://private-registry.example.com:8080/mke`. The setting must always start with `oci://`, and it must not end with a slash `/` . If you uploaded the bundle as previously described, the registry address and path will be the same for chart and image registry, with the only difference being the `oci://` prefix in the chart registry URL. `.spec.registries.chartRegistry.URL` can only be configured once during the first running of the `mkectl apply` command.
`.spec.registries.imageRegistry.caData`	PEM encoded certificate of the Certificate Authority that issued image registry TLS certificates. Optional. Must be provided if registry is using TLS certs issued by a non-publicly trusted CA.
`.spec.registries.chartRegistry.caData`	PEM encoded certificate of the Certificate Authority that issued chart registry TLS certificates. Optional. Must be provided if registry is using TLS certs issued by a non-publicly trusted CA.
`.spec.airgap.enabled = true`	Indicates that your environment is air-gapped.

Example:

spec:
  airgap:
    enabled: true
  registries:
    chartRegistry:
      url: oci://private-registry.example.com:8080/mke
      caData: |-
        -----BEGIN CERTIFICATE-----
        MIIFejCCA2KgAwIBAgIUGUBk1RMNa2X/dijEzY0OwxnuFHowDQYJKoZIhvcNAQEL
        ...
        2+yWdp5K3LBV/+OtCEA=
        -----END CERTIFICATE-----
    imageRegistry:
      url: private-registry.example.com:8080/mke
      caData: |-
        -----BEGIN CERTIFICATE-----
        MIIFejCCA2KgAwIBAgIUGUBk1RMNa2X/dijEzY0OwxnuFHowDQYJKoZIhvcNAQEL
        ...
        2+yWdp5K3LBV/+OtCEA=
        -----END CERTIFICATE-----

Run the mkectl apply command.

Note

If mkectl apply stops before it reports that MKE 4 installed successfully, run mkectl reset --force -f mke4.yaml before you run mkectl apply again. For more information, refer to Create a cluster.

Configure authenticated registry#

If your private registry requires authentication to pull images, you need to provide MKE 4 with pull credentials. These credentials are stored in a kubernetes.io/dockerconfigjson Secret in the mke namespace and are referenced from the MKE4.yaml configuration file. Note that the credentials never appear in the MkeConfig resource itself.

Configure the registry Secret name. To do this, add credentialsSecretName to each registry in the mke4.yaml configuration file that requires authentication:
```
spec:
  airgap:
    enabled: true
  registries:
    imageRegistry:
      url: private-registry.example.com:8080/mke
      credentialsSecretName: mke-registry-creds
    chartRegistry:
      url: oci://private-registry.example.com:8080/mke
      credentialsSecretName: mke-registry-creds
```
When the image and chart registries share the same host, reference the same credentialsSecretName for both, as illustrated above. When they are on different hosts, give each its own Secret name.
Supply the credentials at the time of initial application.

The first time you run the mkectl apply command, pass the credentials with the registry flags. mkectl creates the dockerconfigjson Secret that is named by credentialsSecretName and writes the matching authentication entry into the containerd configuration on each node.
```
mkectl apply \
--image-registry-username <username> --image-registry-password <password> \
--chart-registry-username <username> --chart-registry-password <password>
```
Password Protection

To assure that passwords are not included in your shell history and the process argument list, set them instead in the environment. When a password flag is omitted, mkectl reads it from the corresponding environment variable:
```
export MKE_IMAGE_REGISTRY_PASSWORD=<password>
export MKE_CHART_REGISTRY_PASSWORD=<password>
mkectl apply \
--image-registry-username <username> \
--chart-registry-username <username>
```
Supply the credentials during restore

The mkectl restore command accepts the same four registry flags as mkectl apply. Pass them whenever you restore a cluster that pulls from an authenticated registry:
```
mkectl restore --hosts-path <path_to_hosts_file> --name <backup_tar_file> \
--image-registry-username <username> --image-registry-password <password> \
--chart-registry-username <username> --chart-registry-password <password>
```
During restore, mkectl rebuilds the bootstrap node and pulls from both the image registry and the chart registry. The cluster does not yet exist at that point, so mkectl cannot read the credentials from the Secret named by credentialsSecretName, as it does on a subsequent mkectl apply. When you omit a password flag, mkectl reads the password from the MKE_IMAGE_REGISTRY_PASSWORD or MKE_CHART_REGISTRY_PASSWORD environment variable, as it does for mkectl apply.

Registry authentication adheres to the following rules:

The registry must be served over HTTPS. Because mkectl always connects to the registry over TLS and never transmits credentials over plain HTTP, you cannot use an authenticated registry at an http:// address. The --registry-insecure-skip-tls-verify flag skips certificate verification only. It does not switch the connection to plain HTTP.
Each username and its password must be supplied together.
Supplying --image-registry-username/--image-registry-password requires that spec.registries.imageRegistry.credentialsSecretName be set in the mke4.yaml configuration file, and likewise for the chart registry. A password without a Secret name is rejected.
If the image and chart registries share a host or reference the same credentialsSecretName, their credentials must be identical. containerd stores one credential set per host, and a single Secret holds one credential set.
On any subsequent application of the mkectl apply command against the existing cluster, omit the flags. mkectl reads the stored credentials back from the referenced Secret.
If your registry also uses TLS certificates issued by a non-public CA, set caData on the registry in addition to credentialsSecretName.

Note

The registry-side REGISTRY_USERNAME / REGISTRY_PASSWORD variables in the Preparation section authenticate the skopeo push that uploads the bundle. The credentialsSecretName Secret and the --*-registry-* flags described here authenticate the cluster pulls at install time. The two are independent and may make use of different accounts.

Upgrade#

To upgrade MKE 4 in an air gap environment:

Prepare the new MKE 4 release.
Obtain the pertinent release matrix from the offline bundle, or download it from the official release matrix.

Warning

Do not modify the release matrix. Mirantis does not support release matrixes that have been modified.
Back up the cluster using the mkectl binary that currently manages it. For more information, refer to Perform an MKE 4 to MKE 4+ Upgrade.

Keep this mkectl binary until the upgrade succeeds. If the automatic rollback does not complete, you need it to restore the backup.
Verify that you have the latest version of the mkectl binary. For information on how to download and install mkectl, refer to Install the MKE 4 CLI (mkectl).

Run the upgrade.

mkectl upgrade --upgrade-version <version_to_upgrade_to> \
               --release-matrix <path_to_release_matrix>

After the upgrade concludes, delete the pre-upgrade backup that mkectl upgrade wrote to the directory from which you ran the command.

Windows worker considerations#

If your offline installation includes Windows workers, prepare the environment for both operating systems.

In addition to the standard offline installation requirements:

Upload the windows specific bundle to your registry.
Ensure that the private registry is reachable from both the Linux and Windows nodes.
Ensure that the Linux and Windows nodes both trust the certificate authority in use by the private registry.

MKE 4 versus MKE 3#

MKE 3 requires the use of the docker load command to load offline bundles directly into Docker on every cluster node. While this approach does not require you to have a private registry, it also means that the cluster cannot re-pull the image should any of the loaded images go missing. As such, MKE 3 users must disable Kubernetes garbage collection, which can sometimes prune images of optional components that are not always enabled. This is not an issue with MKE 4, as images are pulled from a private registry that the customer provides, and thus there is no need to disable Kubernetes garbage collection. That said, though, users must ensure that the registry is available at all times and that it is accessible from every cluster node.