Configure TLS certificates for cluster applications¶

Technology Preview

The Container Cloud web UI and StackLight endpoints are available through Transport Layer Security (TLS) with self-signed certificates generated by the Container Cloud provider.

Caution

The Container Cloud endpoints are available only through HTTPS.

Supported applications for TLS certificates configuration¶
Application name	Cluster Type	Comment
Container Cloud web UI	Management
`iam-proxy`	Management and managed	Available since Container Cloud 2.22.0 (Cluster release 11.6.0).
Keycloak	Management
`mcc-cache`	Management

Caution

The organization administrator must ensure that the application host name is resolvable within and outside the cluster.

Caution

Custom TLS certificates for Keycloak are supported for new and existing clusters originally deployed using MOS 21.3 or later.

Workflow of custom MKE certificates configuration¶

Available since 2.24.0 (Cluster releases 14.0.0, 14.0.1) Applies to management clusters only

When you add custom MKE certificates on a management cluster, the following workflow applies:

LCM agents are notified to connect to the management cluster using a different certificate.
After all agents confirm that they are ready to support both current and custom authentication, new MKE certificates apply.
LCM agents switch to the new configuration as soon as it gets valid.
The next cluster reconciliation reconfigures helm-controller for each managed cluster created within the configured management cluster.
If MKE certificates apply to the management cluster, the Container Cloud web UI reconfigures.

Caution

The Container Cloud web UI requires up to 10 minutes to update the MKE certificate configuration for communication with the management cluster. During this time, requests to the management cluster fail with the following example error:

Data loading failed
Failed to get projects list. Server response code: 502

This error is expected and disappears once new certificates apply.

Warning

During certificates application, LCM agents from every node must confirm that they have a new configuration prepared. If managed clusters contain a big number of nodes and some are stuck or orphaned, then the whole process gets stuck. Therefore, before applying new certificates, make sure that all nodes are ready.

Warning

If you apply MKE certificates to the management cluster with proxy enabled, all nodes and pods of this cluster and its managed clusters are triggered for reconfiguration and restart, which may cause the API and workload outage.

Prepare TLS certificates¶

Obtain your DNS server name. For example, container-cloud-auth.example.com.
Buy or generate a certificate from a certification authority (CA) that contains the following items:
- A full CA bundle including the root and all intermediate CA certificates.
- Your server certificate issued for the container-cloud-auth.example.com DNS name.
- Your secret key that was used to sign the certificate signing request. For example, cert.key.
Select the root CA certificate from your CA bundle and add it to root_ca.crt.
Combine all certificates including the root CA, intermediate CA from the CA bundle, and your server certificate into one file. For example, full_chain_cert.crt.

Configure TLS certificates using the Container Cloud web UI¶

Available since MCC 2.24.0 (14.0.0, 14.0.1)

Log in to the Container Cloud web UI with the m:kaas:namespace@operator or m:kaas:namespace@writer permissions.
Switch to the required project using the Switch Project action icon located on top of the main left-side navigation panel.
In the Clusters tab, click the More action icon in the last column of the required cluster and select Configure cluster.
In the Security > TLS Certificates section, click Add certificate.

In the wizard that opens, fill out and save the form:

Parameter	Description
Server name	Host name of the application.
Applications	Drop-down list of available applications for TLS certificates configuration.
Server certificate	Certificate to authenticate the identity of the server to a client. You can also add a valid certificate bundle. The server certificate must be on the top of the chain.
Private key	Private key for the server that must correspond to the public key used in the server certificate.
CA Certificate	CA certificate that issued the server certificate. Required when configuring Keycloak or `mcc-cache`. Use the top-most intermediate certificate if the CA certificate is unavailable.

The Security section displays the expiration date and the readiness status for every application with user-defined certificates.

Optional. Edit the certificate using the Edit action icon located to the right of the application status and edit the form filled out in the previous step.

Note

To revoke a certificate, use the Delete action icon located to the right of the application status.

Configure TLS certificates using the Container Cloud API¶

For clusters originally deployed using MOS release earlier than 21.3, download the latest version of the bootstrap script on the management cluster:
```
wget https://binary.mirantis.com/releases/get_container_cloud.sh
chmod 0755 get_container_cloud.sh
./get_container_cloud.sh
```
Change the directory to kaas-boostrap.

If you deleted this directory, restore it using the step 1 of the procedure described in Container Cloud documentation: Collect cluster logs.

Select from the following options:

Set a TLS certificate for the Container Cloud web UI:

./container-cloud set certificate \
  --cert-file <fullPathToCertForUI> \
  --key-file <pathToPrivateKeyForUI> \
  --for ui \
  --hostname  <applicationHostName> \
  --kubeconfig <mgmtClusterKubeconfig>

Since Container Cloud 2.22.0 (Cluster release 11.6.0), set a TLS certificate for iam-proxy:

./container-cloud set certificate \
  --cert-file <fullPathToCertForIAMProxyEndpoint> \
  --key-file <pathToPrivateKeyForIAMProxyEndpoint> \
  --for <IAMProxyEndpoint> --hostname <IAMProxyEndpointHostName> \
  --kubeconfig <mgmtClusterKubeconfig> \
  --cluster-name <targetClusterName> \
  --cluster-namespace <targetClusterNamespace>

Possible values for IAMProxyEndpoint are as follows:

iam-proxy-alerta
iam-proxy-alertmanager
iam-proxy-grafana
iam-proxy-kibana
iam-proxy-prometheus

Set a TLS certificate for Keycloak:

./container-cloud set certificate \
  --cacert-file <fullRootpathToCACertForKeycloak> \
  --cert-file <fullPathToCertForKeycloak> \
  --key-file <pathToPrivateKeyForKeycloak> \
  --for keycloak --hostname <applicationHostName> \
  --kubeconfig <mgmtClusterKubeconfig>

Set a TLS certificate for mcc-cache:

./container-cloud set certificate \
  --cacert-file <fullRootpathToCACertForCache> \
  --cert-file <fullPathToCertForCache> \
  --key-file <pathToPrivateKeyForCache> \
  --for cache --hostname <applicationHostName> \
  --kubeconfig <mgmtClusterKubeconfig> \
  --cluster-name <targetClusterName> \
  --cluster-namespace <targetClusterProjectName>

Caution

All managed clusters must be updated to the latest available Cluster release.

Caution

The organization administrator must ensure that the mcc-cache host name is resolvable for all managed clusters.

In the commands above, replace the parameters enclosed in angle brackets with the corresponding values of your cluster.

Flag	Description
`--cacert-file`	Must contain only one PEM-encoded root CA certificate in the certificate chain of trust.
`--cert-file`	Must contain all certificates in the server certificate chain of trust including the PEM-encoded server certificate.
`--key-file`	Private key used to generate the provided certificate.
`--for` `<applicationName>` or `<IAMProxyEndpoint>`	Configures a certificate for a supported application. The list of possible values for application names includes: `cache`, `keycloak`, or `ui`.
`--hostname`	DNS server host name.
`--kubeconfig`	Management cluster `kubeconfig` that is by default located in the `kaas-bootstrap` directory.
`--cluster-name`	Target cluster name.
`--cluster-namespace`	Project name of the target cluster.

Example command:

./container-cloud set certificate \
  --cacert-file root_ca.crt \
  --cert-file full_chain_cert.crt \
  --key-file cert.key \
  --for keycloak \
  --hostname container-cloud-auth.example.com \
  --kubeconfig kubeconfig

The self-signed certificates generated and managed by the Container Cloud provider are stored in *-tls-certs secrets in the kaas and stacklight namespaces.

Renew expired TLS certificates¶

MOSK provides automatic renewal of certificates for internal Container Cloud services. Custom certificates require manual renewal.

If you have permissions to view the default project in the Container Cloud web UI, you may see the Certificate Is Expiring Soon warning for custom certificates. The warning appears on top of the Container Cloud web UI. It displays the certificate with the least number of days before expiration. Click See Details and get more information about other expiring certificates. You can also find the details about the expiring certificates in the Status column’s Certificate Issues tooltip on the Clusters page.

The Certificate Issues status may include the following messages:

Some certificates require manual renewal
A custom certificate is expiring in less than seven days. Renew the certificate manually using the same container-cloud binary as for the certificate configuration. For details, see Configure TLS certificates using the Container Cloud API.
Some certificates were not renewed automatically
An automatic certificate renewal issue. Unexpected error, contact Mirantis support.