Configure TLS certificates for cluster applications¶

Technology Preview

The Container Cloud web UI and StackLight endpoints are available through Transport Layer Security (TLS) with self-signed certificates generated by the Container Cloud provider.

Caution

The Container Cloud endpoints are available only through HTTPS.

Supported applications for TLS certificates configuration¶
Application name	Cluster Type	Comment
Container Cloud web UI	Management
`iam-proxy`	Management, regional, and managed	Available since Container Cloud 2.22.0.
Keycloak	Management
`mcc-cache`	Management and regional
MKE	Management, regional, and managed	Available for clusters deployed or updated by Container Cloud using the latest Cluster release in its series. For management and regional clusters, available since Container Cloud 2.24.0. Caution For MKE clusters that are part of MOSK infrastructure, the feature is not supported yet.

Caution

The organization administrator must ensure that the application host name is resolvable within and outside the cluster.
Custom TLS certificates for Keycloak are supported for new and existing clusters originally deployed using Container Cloud 2.9.0 or later.

Workflow of custom MKE certificates configuration¶

Available since 2.24.0 Applies to management and regional clusters only

When you add custom MKE certificates on management and regional clusters, the following workflow applies:

The regional cluster notifies all LCM agents in all clusters of the region that the regional cluster will be accessible by a different address and the cluster identity needs to be verified using a different certificate.
After all agents confirm that they are ready to support both current and custom authentication, new MKE certificates apply.
LCM agents switch to the new configuration as soon as it gets valid.
The next cluster reconciliation reconfigures helm-controller for each cluster in the region.
If MKE certificates apply to the management cluster, the Container Cloud web UI reconfigures.

Caution

If MKE certificates apply to the management cluster, the Container Cloud web UI requires up to 10 minutes to update the configuration for communication with the management cluster. During this time, requests to the management cluster fail with the following example error:

Data loading failed
Failed to get projects list. Server response code: 502

This error is expected and disappears once new certificates apply.

Warning

During certificates application, LCM agents from every node must confirm that they have a new configuration prepared. If the region contains a big number of nodes and some are stuck or orphaned, then the whole process gets stuck. Therefore, before applying new certificates, make sure that all nodes of the region are ready.

Prepare TLS certificates¶

Obtain your DNS server name. For example, container-cloud-auth.example.com.
Buy or generate a certificate from a certification authority (CA) that contains the following items:
- A full CA bundle including the root and all intermediate CA certificates.
- Your server certificate issued for the container-cloud-auth.example.com DNS name.
- Your secret key that was used to sign the certificate signing request. For example, cert.key.
Select the root CA certificate from your CA bundle and add it to root_ca.crt.
Combine all certificates including the root CA, intermediate CA from the CA bundle, and your server certificate into one file. For example, full_chain_cert.crt.

Configure TLS certificates using the Container Cloud web UI¶

Available since 2.24.0

Log in to the Container Cloud web UI with the m:kaas:namespace@operator or m:kaas:namespace@writer permissions.
Switch to the required project using the Switch Project action icon located on top of the main left-side navigation panel.
In the Clusters tab, click the More action icon in the last column of the required cluster and select Configure cluster.
In the Security > TLS Certificates section, click Add certificate.

In the wizard that opens, fill out and save the form:

Parameter	Description
Server name	Host name of the application.
Applications	Drop-down list of available applications for TLS certificates configuration.
Server certificate	Certificate to authenticate the identity of the server to a client. You can also add a valid certificate bundle. The server certificate must be on the top of the chain.
Private key	Private key for the server that must correspond to the public key used in the server certificate.
CA Certificate	CA certificate that issued the server certificate. Required when configuring Keycloak, `mcc-cache`, or MKE. Use the top-most intermediate certificate if the CA certificate is unavailable.

The Security section displays the expiration date and the readiness status for every application with user-defined certificates.

Optional. Edit the certificate using the Edit action icon located to the right of the application status and edit the form filled out in the previous step.

Note

To revoke a certificate, use the Delete action icon located to the right of the application status.

Configure TLS certificates using the Container Cloud API¶

For clusters deployed using the Container Cloud release earlier than 2.9.0, download the latest version of the bootstrap script on the management cluster:
```
wget https://binary.mirantis.com/releases/get_container_cloud.sh
chmod 0755 get_container_cloud.sh
./get_container_cloud.sh
```
Change the directory to kaas-boostrap.

If you deleted this directory, restore it using the step 1 of the Collect cluster logs procedure.

Select from the following options:

Set a TLS certificate for the Container Cloud web UI:

./container-cloud set certificate \
  --cert-file <fullPathToCertForUI> \
  --key-file <pathToPrivateKeyForUI> \
  --for ui \
  --hostname  <applicationHostName> \
  --kubeconfig <mgmtClusterKubeconfig>

Since Container Cloud 2.22.0, set a TLS certificate for iam-proxy:

./container-cloud set certificate \
  --cert-file <fullPathToCertForIAMProxyEndpoint> \
  --key-file <pathToPrivateKeyForIAMProxyEndpoint> \
  --for <IAMProxyEndpoint> --hostname <IAMProxyEndpointHostName> \
  --kubeconfig <mgmtClusterKubeconfig> \
  --cluster-name <targetClusterName> \
  --cluster-namespace <targetClusterNamespace>

Possible values for IAMProxyEndpoint are as follows:

iam-proxy-alerta
iam-proxy-alertmanager
iam-proxy-grafana
iam-proxy-kibana
iam-proxy-prometheus

Set a TLS certificate for Keycloak:

./container-cloud set certificate \
  --cacert-file <fullRootpathToCACertForKeycloak> \
  --cert-file <fullPathToCertForKeycloak> \
  --key-file <pathToPrivateKeyForKeycloak> \
  --for keycloak --hostname <applicationHostName> \
  --kubeconfig <mgmtClusterKubeconfig>

Set a TLS certificate for mcc-cache:

./container-cloud set certificate \
  --cacert-file <fullRootpathToCACertForCache> \
  --cert-file <fullPathToCertForCache> \
  --key-file <pathToPrivateKeyForCache> \
  --for cache --hostname <applicationHostName> \
  --kubeconfig <mgmtClusterKubeconfig> \
  --cluster-name <targetClusterName> \
  --cluster-namespace <targetClusterProjectName>

Caution

All managed clusters in the region must be updated to the latest available Cluster release.
The organization administrator must ensure that the mcc-cache host name is resolvable for all managed clusters in the region.

Set a TLS certificate for MKE:

./container-cloud set certificate \
  --cacert-file <fullRootpathToCACertForMKE> \
  --cert-file <fullPathToCertForMKE> \
  --key-file <pathToPrivateKeyForMKE> \
  --for mke --hostname <applicationHostName> \
  --kubeconfig <mgmtClusterKubeconfig> \
  --cluster-name <targetClusterName> \
  --cluster-namespace <targetClusterNamespace>

Caution

For MKE clusters that are part of MOSK infrastructure, the feature is not supported yet.

In the commands above, replace the parameters enclosed in angle brackets with the corresponding values of your cluster.

Flag	Description
`--cacert-file`	Must contain only one PEM-encoded root CA certificate in the certificate chain of trust.
`--cert-file`	Must contain all certificates in the server certificate chain of trust including the PEM-encoded server certificate.
`--key-file`	Private key used to generate the provided certificate.
`--for` `<applicationName>` or `<IAMProxyEndpoint>`	Configures a certificate for a supported application. The list of possible values for application names includes: `cache`, `keycloak`, `mke`, `ui`.
`--hostname`	DNS server host name.
`--kubeconfig`	Management cluster `kubeconfig` that is by default located in the `kaas-bootstrap` directory.
`--cluster-name`	Target cluster name.
`--cluster-namespace`	Target cluster project name in Container Cloud.

Example command:

./container-cloud set certificate \
  --cacert-file root_ca.crt \
  --cert-file full_chain_cert.crt \
  --key-file cert.key \
  --for keycloak \
  --hostname container-cloud-auth.example.com \
  --kubeconfig kubeconfig

The self-signed certificates generated and managed by the Container Cloud provider are stored in *-tls-certs secrets in the kaas and stacklight namespaces.

Renew expired TLS certificates¶

Container Cloud provides automatic renewal of certificates for internal Container Cloud services and for MKE on managed clusters deployed by Container Cloud. Custom certificates require manual renewal.

If you have permissions to view the default project in the Container Cloud web UI, you may see the Certificate Is Expiring Soon warning for custom certificates. The warning appears on top of the Container Cloud web UI. It displays the certificate with the least number of days before expiration. Click See Details and get more information about other expiring certificates. You can also find the details about the expiring certificates in the Status column’s Certificate Issues tooltip on the Clusters page.

The Certificate Issues status may include the following messages:

Some certificates require manual renewal
A custom certificate is expiring in less than seven days. Renew the certificate manually using the same container-cloud binary as for the certificate configuration. For details, see Configure TLS certificates using the Container Cloud API.
Some certificates were not renewed automatically
An automatic certificate renewal issue. Unexpected error, contact Mirantis support.