Configure TLS certificates for cluster applications

TechPreview

This section describes supported applications for Transport Layer Security (TLS) certificates configuration. By default, application endpoints are available through TLS with self-signed certificates generated by the MOSK management provider.

Caution

The MOSK management endpoints are available only through HTTPS.

Supported applications for TLS certificates configuration
Application name	Cluster Type
`iam-proxy`	Management and MOSK
k0s [0]	Management
Keycloak	Management
`mcc-cache`	Management
MOSK management console	Management

Important

During the update to MOSK management 2.32.0, the management cluster migrates from MKE to k0s. Any custom MKE TLS configuration in the Cluster object is automatically moved from tls.mke to tls.k0s. After the update, configure Kubernetes API certificates through tls.k0s only.

Caution

The organization administrator must ensure that the application host name is resolvable within and outside the cluster.

Note

Custom TLS certificates for Keycloak are supported for new and existing clusters originally deployed using MOS 21.3 or later.

Warning

Switching to a custom k0s certificate is irreversible. You can update the certificate at any time, but you cannot revert the Kubernetes API endpoint to the self-signed certificate generated by the MOSK management provider. An attempt to remove the tls.k0s section from the Cluster object fails with the unsetting k0s tls spec is not allowed error.

Workflow of custom k0s certificates configuration

Applies to management clusters only

When you add custom k0s certificates to a management cluster, the following workflow applies:

The certificate and private key are placed on every control plane node of the cluster.
After all control plane nodes report that the certificate is in place, the Kubernetes API server is reconfigured to serve it for the configured host name.
Control plane nodes restart the Kubernetes API server one by one to apply the new configuration.
The configured host name becomes the Kubernetes API endpoint of the cluster.

The custom certificate is served through Server Name Indication (SNI) for the configured host name only. Requests that reach the Kubernetes API through the load-balancer IP address, as well as in-cluster endpoints, keep using the self-signed certificate generated by the MOSK management provider. Therefore, existing kubeconfig files that refer to the load-balancer IP address remain valid.

Caution

Ensure that the configured host name resolves to the cluster API load-balancer address both on the cluster nodes and inside the cluster pod network.

Warning

Applying custom k0s certificates restarts the Kubernetes API server on control plane nodes one by one. Each control plane node is cordoned and drained in turn, so the pods running on it are rescheduled. The cluster API remains available through the load balancer, which keeps serving the requests from the remaining control plane nodes. Before applying new certificates, make sure that all cluster nodes are ready.

Prepare TLS certificates

Obtain your DNS server name. For example, container-cloud-auth.example.com.
Buy or generate a certificate from a certification authority (CA) that contains the following items:
- A full CA bundle including the root and all intermediate CA certificates.
- Your server certificate issued for the container-cloud-auth.example.com DNS name.
- Your secret key that was used to sign the certificate signing request. For example, cert.key.
Select the root CA certificate from your CA bundle and add it to root_ca.crt.
Combine all certificates including the root CA, intermediate CA from the CA bundle, and your server certificate into one file. For example, full_chain_cert.crt.

Configure TLS certificates using the MOSK management console

Log in to the MOSK management console with the m:kaas:namespace@operator or m:kaas:namespace@writer permissions.
Switch to the required project using the Switch Project action icon located on top of the main left-side navigation panel.
In the Clusters tab, click the More action icon in the last column of the required cluster and select Configure cluster.
In the Security > TLS Certificates section, click Add certificate.

In the wizard that opens, fill out and save the form:

Parameter	Description
Server name	Host name of the application.
Applications	Drop-down list of available applications for TLS certificates configuration.
Server certificate	Certificate to authenticate the identity of the server to a client. You can also add a valid certificate bundle. The server certificate must be on the top of the chain.
Private key	Private key for the server that must correspond to the public key used in the server certificate.
CA Certificate	CA certificate that issued the server certificate. Required when configuring Keycloak, `mcc-cache`, or k0s. Use the top-most intermediate certificate if the CA certificate is unavailable.

The Security section displays the expiration date and the readiness status for every application with user-defined certificates.

Optional. Edit the certificate using the Edit action icon located to the right of the application status and edit the form filled out in the previous step.

Note

To revoke a certificate, use the Delete action icon located to the right of the application status. Certificates configured for k0s cannot be revoked.
Strongly recommended. Back up the Kubernetes underlay as described in Create backups of the Kubernetes underlay.

Since the procedure above modifies the cluster configuration, a fresh backup is required to restore the cluster in case further reconfigurations fail.

Important

Because the restoration process is complicated, we strongly recommend contacting Mirantis support for assistance.
Important

If you still decide to restore the Kubernetes underlay from a backup on your own, you must scale down helm-controller on the cluster being restored if the Kubernetes underlay version of the affected cluster after the restore will differ from the Kubernetes underlay version in the ClusterRelease object that is set in Cluster objects in the management cluster:
- If you are restoring k0s on a management cluster: before starting the restore, scale down helm-controller on each affected MOSK cluster. This prevents unintended Ceph and OpenStack downgrades on MOSK clusters after the management cluster is restored.
- If you are restoring MKE on a MOSK cluster: immediately after the restore completes, scale down helm-controller. Because the restore rolls the cluster back to an older release, this prevents it from triggering a premature upgrade of Helm releases.

Configure TLS certificates using the MOSK management API

For clusters originally deployed using MOS release earlier than 21.3, download the latest version of the bootstrap script on the management cluster:
```
wget https://binary.mirantis.com/releases/get_container_cloud.sh
chmod 0755 get_container_cloud.sh
./get_container_cloud.sh
```
Change the directory to kaas-boostrap.

If you deleted this directory, restore it using the step 1 of the procedure described in Collect cluster logs.

Select one of the following options:

Set a TLS certificate for the MOSK management console:

./container-cloud set certificate \
  --cert-file <fullPathToCertForUI> \
  --key-file <pathToPrivateKeyForUI> \
  --for ui \
  --hostname  <applicationHostName> \
  --kubeconfig <mgmtClusterKubeconfig>

Set a TLS certificate for iam-proxy:

./container-cloud set certificate \
  --cert-file <fullPathToCertForIAMProxyEndpoint> \
  --key-file <pathToPrivateKeyForIAMProxyEndpoint> \
  --for <IAMProxyEndpoint> --hostname <IAMProxyEndpointHostName> \
  --kubeconfig <mgmtClusterKubeconfig> \
  --cluster-name <targetClusterName> \
  --cluster-namespace <targetClusterNamespace>

Possible values for IAMProxyEndpoint are as follows:

iam-proxy-alerta
iam-proxy-alertmanager
iam-proxy-grafana
iam-proxy-kibana
iam-proxy-prometheus

Set a TLS certificate for Keycloak:

./container-cloud set certificate \
  --cacert-file <fullRootpathToCACertForKeycloak> \
  --cert-file <fullPathToCertForKeycloak> \
  --key-file <pathToPrivateKeyForKeycloak> \
  --for keycloak --hostname <applicationHostName> \
  --kubeconfig <mgmtClusterKubeconfig>

Set a TLS certificate for mcc-cache:

./container-cloud set certificate \
  --cacert-file <fullRootpathToCACertForCache> \
  --cert-file <fullPathToCertForCache> \
  --key-file <pathToPrivateKeyForCache> \
  --for cache --hostname <applicationHostName> \
  --kubeconfig <mgmtClusterKubeconfig> \
  --cluster-name <targetClusterName> \
  --cluster-namespace <targetClusterProjectName>

Caution

The organization administrator must ensure that the mcc-cache host name is resolvable for all MOSK clusters.

Set a TLS certificate for k0s:
```
./container-cloud set certificate \
  --cacert-file <fullRootpathToCACertForK0s> \
  --cert-file <fullPathToCertForK0s> \
  --key-file <pathToPrivateKeyForK0s> \
  --for k0s \
  --hostname <applicationHostName> \
  --kubeconfig <mgmtClusterKubeconfig> \
  --cluster-name <targetClusterName> \
  --cluster-namespace <targetClusterProjectName>
```
Caution

All management clusters must be updated to the latest available Cluster release.

Note

The command resolves the value of the --hostname on the machine where you run it and fails with the hostname <applicationHostName> cannot be resolved error if the host name cannot be resolved there.

Note

The command returns once the certificate is uploaded to the cluster, before it is applied on the control plane nodes. For the remaining steps and their impact, see Workflow of custom k0s certificates configuration.

In the commands above, replace the parameters enclosed in angle brackets with the corresponding values of your cluster.

Flag	Description
`--cacert-file`	Must contain only one PEM-encoded root CA certificate in the certificate chain of trust.
`--cert-file`	Must contain all certificates in the server certificate chain of trust including the PEM-encoded server certificate.
`--key-file`	Private key used to generate the provided certificate.
`--for` `<applicationName>` or `<IAMProxyEndpoint>`	Configures a certificate for a supported application. The list of possible values for application names includes: `cache`, `k0s`, `keycloak`, or `ui`.
`--hostname`	DNS server host name.
`--kubeconfig`	Management cluster `kubeconfig` that is by default located in the `kaas-bootstrap` directory.
`--cluster-name`	Target cluster name.
`--cluster-namespace`	Project name of the target cluster.

Example command:

./container-cloud set certificate \
  --cacert-file root_ca.crt \
  --cert-file full_chain_cert.crt \
  --key-file cert.key \
  --for keycloak \
  --hostname container-cloud-auth.example.com \
  --kubeconfig kubeconfig

Strongly recommended. Back up the Kubernetes underlay as described in Create backups of the Kubernetes underlay.

Since the procedure above modifies the cluster configuration, a fresh backup is required to restore the cluster in case further reconfigurations fail.

Important

Because the restoration process is complicated, we strongly recommend contacting Mirantis support for assistance.
Important

If you still decide to restore the Kubernetes underlay from a backup on your own, you must scale down helm-controller on the cluster being restored if the Kubernetes underlay version of the affected cluster after the restore will differ from the Kubernetes underlay version in the ClusterRelease object that is set in Cluster objects in the management cluster:
- If you are restoring k0s on a management cluster: before starting the restore, scale down helm-controller on each affected MOSK cluster. This prevents unintended Ceph and OpenStack downgrades on MOSK clusters after the management cluster is restored.
- If you are restoring MKE on a MOSK cluster: immediately after the restore completes, scale down helm-controller. Because the restore rolls the cluster back to an older release, this prevents it from triggering a premature upgrade of Helm releases.

The self-signed certificates generated and managed by the MOSK management provider are stored in *-tls-certs secrets in the kaas and stacklight namespaces.

Renew expired TLS certificates

MOSK provides automatic renewal of certificates for internal MOSK services. Custom certificates require manual renewal.

If you have permissions to view the default project in the MOSK management console, you may see the Certificate Is Expiring Soon warning for custom certificates. The warning appears on top of the MOSK management console. It displays the certificate with the least number of days before expiration. Click See Details and get more information about other expiring certificates. You can also find the details about the expiring certificates in the Status column’s Certificate Issues tooltip on the Clusters page.

The Certificate Issues status may include the following messages:

Some certificates require manual renewal
A custom certificate is expiring in less than seven days. Renew the certificate manually using the same container-cloud binary as for the certificate configuration. For details, see Configure TLS certificates using the MOSK management API.
Some certificates were not renewed automatically
An automatic certificate renewal issue. Unexpected error, contact Mirantis support.

Caution

After certificate renewal, depending on the cluster type, back up MKE or k0s as described in Configure backups of the Kubernetes underlay.