Configure TLS certificates for cluster applications

Technology Preview

The Container Cloud web UI and StackLight endpoints are available through Transport Layer Security (TLS) with self-signed certificates generated by the Container Cloud provider.

Caution

The Container Cloud endpoints are available only through HTTPS.

Supported applications for TLS certificates configuration

Application name

Cluster Type

Comment

Container Cloud web UI

Management

iam-proxy

Management and managed

Available since Container Cloud 2.22.0.

Keycloak

Management

mcc-cache

Management

MKE

Management and managed

Available for clusters deployed or updated by Container Cloud using the latest Cluster release in its series.

For management clusters, available since Container Cloud 2.24.0.

Note

For attached MKE clusters that were not originally deployed by Container Cloud, the feature is not tested on the system integration level. Therefore, Mirantis does not recommend using the feature on production deployments.

Caution

For MKE clusters that are part of MOSK infrastructure, the feature is not supported yet.

Caution

  • The organization administrator must ensure that the application host name is resolvable within and outside the cluster.

  • Custom TLS certificates for Keycloak are supported for new and existing clusters originally deployed using Container Cloud 2.9.0 or later.

Workflow of custom MKE certificates configuration

Available since 2.24.0 Applies to management clusters only

When you add custom MKE certificates on a management cluster, the following workflow applies:

  1. LCM agents are notified to connect to the management cluster using a different certificate.

  2. After all agents confirm that they are ready to support both current and custom authentication, new MKE certificates apply.

  3. LCM agents switch to the new configuration as soon as it gets valid.

  4. The next cluster reconciliation reconfigures helm-controller for each managed cluster created within the configured management cluster.

  5. If MKE certificates apply to the management cluster, the Container Cloud web UI reconfigures.

Caution

If MKE certificates apply to the management cluster, the Container Cloud web UI requires up to 10 minutes to update the configuration for communication with the management cluster. During this time, requests to the management cluster fail with the following example error:

Data loading failed
Failed to get projects list. Server response code: 502

This error is expected and disappears once new certificates apply.

Warning

During certificates application, LCM agents from every node must confirm that they have a new configuration prepared. If managed clusters contain a big number of nodes and some are stuck or orphaned, then the whole process gets stuck. Therefore, before applying new certificates, make sure that all nodes are ready.

Warning

If you apply MKE certificates to the management cluster with proxy enabled, all nodes and pods of this cluster and its managed clusters are triggered for reconfiguration and restart, which may cause the API and workload outage.

Prepare TLS certificates

  1. Obtain your DNS server name. For example, container-cloud-auth.example.com.

  2. Buy or generate a certificate from a certification authority (CA) that contains the following items:

    • A full CA bundle including the root and all intermediate CA certificates.

    • Your server certificate issued for the container-cloud-auth.example.com DNS name.

    • Your secret key that was used to sign the certificate signing request. For example, cert.key.

  3. Select the root CA certificate from your CA bundle and add it to root_ca.crt.

  4. Combine all certificates including the root CA, intermediate CA from the CA bundle, and your server certificate into one file. For example, full_chain_cert.crt.

Configure TLS certificates using the Container Cloud web UI

Available since 2.24.0

  1. Log in to the Container Cloud web UI with the m:kaas:namespace@operator or m:kaas:namespace@writer permissions.

  2. Switch to the required project using the Switch Project action icon located on top of the main left-side navigation panel.

  3. In the Clusters tab, click the More action icon in the last column of the required cluster and select Configure cluster.

  4. In the Security > TLS Certificates section, click Add certificate.

  5. In the wizard that opens, fill out and save the form:

    Parameter

    Description

    Server name

    Host name of the application.

    Applications

    Drop-down list of available applications for TLS certificates configuration.

    Server certificate

    Certificate to authenticate the identity of the server to a client. You can also add a valid certificate bundle. The server certificate must be on the top of the chain.

    Private key

    Private key for the server that must correspond to the public key used in the server certificate.

    CA Certificate

    CA certificate that issued the server certificate. Required when configuring Keycloak, mcc-cache, or MKE. Use the top-most intermediate certificate if the CA certificate is unavailable.

    The Security section displays the expiration date and the readiness status for every application with user-defined certificates.

  6. Optional. Edit the certificate using the Edit action icon located to the right of the application status and edit the form filled out in the previous step.

    Note

    To revoke a certificate, use the Delete action icon located to the right of the application status.

Configure TLS certificates using the Container Cloud API

  1. For clusters deployed using the Container Cloud release earlier than 2.9.0, download the latest version of the bootstrap script on the management cluster:

    wget https://binary.mirantis.com/releases/get_container_cloud.sh
    chmod 0755 get_container_cloud.sh
    ./get_container_cloud.sh
    
  2. Change the directory to kaas-boostrap.

    If you deleted this directory, restore it using the step 1 of the Collect cluster logs procedure.

  3. Select from the following options:

    • Set a TLS certificate for the Container Cloud web UI:

      ./container-cloud set certificate \
        --cert-file <fullPathToCertForUI> \
        --key-file <pathToPrivateKeyForUI> \
        --for ui \
        --hostname  <applicationHostName> \
        --kubeconfig <mgmtClusterKubeconfig>
      
    • Since Container Cloud 2.22.0, set a TLS certificate for iam-proxy:

      ./container-cloud set certificate \
        --cert-file <fullPathToCertForIAMProxyEndpoint> \
        --key-file <pathToPrivateKeyForIAMProxyEndpoint> \
        --for <IAMProxyEndpoint> --hostname <IAMProxyEndpointHostName> \
        --kubeconfig <mgmtClusterKubeconfig> \
        --cluster-name <targetClusterName> \
        --cluster-namespace <targetClusterNamespace>
      

      Possible values for IAMProxyEndpoint are as follows:

      • iam-proxy-alerta

      • iam-proxy-alertmanager

      • iam-proxy-grafana

      • iam-proxy-kibana

      • iam-proxy-prometheus

    • Set a TLS certificate for Keycloak:

      ./container-cloud set certificate \
        --cacert-file <fullRootpathToCACertForKeycloak> \
        --cert-file <fullPathToCertForKeycloak> \
        --key-file <pathToPrivateKeyForKeycloak> \
        --for keycloak --hostname <applicationHostName> \
        --kubeconfig <mgmtClusterKubeconfig>
      
    • Set a TLS certificate for mcc-cache:

      ./container-cloud set certificate \
        --cacert-file <fullRootpathToCACertForCache> \
        --cert-file <fullPathToCertForCache> \
        --key-file <pathToPrivateKeyForCache> \
        --for cache --hostname <applicationHostName> \
        --kubeconfig <mgmtClusterKubeconfig> \
        --cluster-name <targetClusterName> \
        --cluster-namespace <targetClusterProjectName>
      

      Caution

      • All managed clusters must be updated to the latest available Cluster release.

      • The organization administrator must ensure that the mcc-cache host name is resolvable for all managed clusters.

    • Set a TLS certificate for MKE:

      ./container-cloud set certificate \
        --cacert-file <fullRootpathToCACertForMKE> \
        --cert-file <fullPathToCertForMKE> \
        --key-file <pathToPrivateKeyForMKE> \
        --for mke --hostname <applicationHostName> \
        --kubeconfig <mgmtClusterKubeconfig> \
        --cluster-name <targetClusterName> \
        --cluster-namespace <targetClusterNamespace>
      

      Caution

      For MKE clusters that are part of MOSK infrastructure, the feature is not supported yet.

    In the commands above, replace the parameters enclosed in angle brackets with the corresponding values of your cluster.

    Flag

    Description

    --cacert-file

    Must contain only one PEM-encoded root CA certificate in the certificate chain of trust.

    --cert-file

    Must contain all certificates in the server certificate chain of trust including the PEM-encoded server certificate.

    --key-file

    Private key used to generate the provided certificate.

    --for <applicationName> or <IAMProxyEndpoint>

    Configures a certificate for a supported application. The list of possible values for application names includes: cache, keycloak, mke, ui.

    --hostname

    DNS server host name.

    --kubeconfig

    Management cluster kubeconfig that is by default located in the kaas-bootstrap directory.

    --cluster-name

    Target cluster name.

    --cluster-namespace

    Target cluster project name in Container Cloud.

    Example command:

    ./container-cloud set certificate \
      --cacert-file root_ca.crt \
      --cert-file full_chain_cert.crt \
      --key-file cert.key \
      --for keycloak \
      --hostname container-cloud-auth.example.com \
      --kubeconfig kubeconfig
    

The self-signed certificates generated and managed by the Container Cloud provider are stored in *-tls-certs secrets in the kaas and stacklight namespaces.

Renew expired TLS certificates

Container Cloud provides automatic renewal of certificates for internal Container Cloud services and for MKE on managed clusters deployed by Container Cloud. Custom certificates require manual renewal.

If you have permissions to view the default project in the Container Cloud web UI, you may see the Certificate Is Expiring Soon warning for custom certificates. The warning appears on top of the Container Cloud web UI. It displays the certificate with the least number of days before expiration. Click See Details and get more information about other expiring certificates. You can also find the details about the expiring certificates in the Status column’s Certificate Issues tooltip on the Clusters page.

The Certificate Issues status may include the following messages:

  • Some certificates require manual renewal

    A custom certificate is expiring in less than seven days. Renew the certificate manually using the same container-cloud binary as for the certificate configuration. For details, see Configure TLS certificates using the Container Cloud API.

  • Some certificates were not renewed automatically

    An automatic certificate renewal issue. Unexpected error, contact Mirantis support.