Cluster

This section describes the Cluster resource used in Mirantis Container Cloud API for the OpenStack-based clusters. The Cluster resource describes the cluster-level parameters.

For demonstration purposes, the Container Cloud Cluster custom resource (CR) can be split into the following major sections:

Warning

The fields of the Cluster resource that are located under the status section including providerStatus are available for viewing only. They are automatically generated by the OpenStack cloud provider and must not be modified using Container Cloud API.

metadata

The Container Cloud Cluster custom resource (CR) contains the following fields:

  • apiVersion

    cluster.k8s.io/v1alpha1

  • kind

    Object type that is Cluster

The metadata object field of the Cluster resource contains the following fields:

  • name

    Cluster name. A managed cluster name is set in the Cluster Name field of the Create Cluster wizard in the Container Cloud web UI. Management and regional cluster names are set in the bootstrap script.

  • namespace

    Namespace in which the Cluster object is created. Management and regional clusters are created in the default namespace. The namespace of a managed cluster matches the selected Project name in the Container Cloud web UI.

  • labels

    Key-value pairs attached to the object:

    • kaas.mirantis.com/provider

      Provider type that is openstack for the OpenStack-based clusters.

    • kaas.mirantis.com/region

      Region name. The default region name for a management cluster is region-one. For regional clusters, it is configurable using the REGION parameter in the bootstrap script.

    Warning

    Labels and annotations that are not documented in this API Reference are generated automatically by Container Cloud. Do not modify them using the Container Cloud API.

Configuration example:

apiVersion: cluster.k8s.io/v1alpha1
kind: Cluster
metadata:
  name: demo
  namespace: test
  labels:
    kaas.mirantis.com/provider: openstack
    kaas.mirantis.com/region: region-one

spec:providerSpec

The providerSpec object field of the Cluster resource contains all required details to create the cluster-level resources. It also contains fields required for LCM deployment and the Container Cloud components integration.

The providerSpec object field is custom for each cloud provider and contains the following generic fields:

  • apiVersion

    openstackproviderconfig.k8s.io/v1alpha1

  • kind

    Object type that is OpenstackClusterProviderSpec

Configuration example:

spec:
  ...
  providerSpec:
    value:
      apiVersion: openstackproviderconfig.k8s.io/v1alpha1
      kind:  OpenstackClusterProviderSpec

spec:providerSpec common

The common providerSpec object field of the Cluster resource contains the following fields:

  • credentials

    Name of the OpenStackCredential object used by the cluster to connect to the provider back end

  • dedicatedControlPlane

    Cluster control plane nodes to be tainted, defaults to true

  • publicKeys

    List of the PublicKey resource references

    • name

      Public key name

  • release

    Name of the ClusterRelease object to install on a cluster

  • helmReleases

    List of enabled Helm releases from the Release object that run on a cluster

  • proxy

    Name of the Proxy object

  • tls

    TLS configuration for endpoints of a cluster

    • keycloak

      KeyCloak endpoint

      • tlsConfigRef

        Reference to the TLSConfig object

    • ui

      Web UI endpoint

      • tlsConfigRef

        Reference to the TLSConfig object

    For more details, see TLSConfig resource.

  • maintenance

    Maintenance mode of a cluster. Prepares a cluster for maintenance and enables the possibility to switch machines into maintenance mode.

  • containerRegistries

    List of the ContainerRegistries resources names.

  • ntpEnabled

    NTP server mode. Boolean, enabled by default.

    Since Container Cloud 2.23.0, you can optionally disable NTP to disable the management of chrony configuration by Container Cloud and use your own system for chrony management. Otherwise, configure the regional NTP server parameters to be applied to all machines of regional and managed clusters in the specified region.

    Before Container Cloud 2.23.0, you can optionally configure NTP parameters if servers from the Ubuntu NTP pool (*.ubuntu.pool.ntp.org) are accessible from the node where a management or regional cluster is being provisioned. Otherwise, this configuration is mandatory.

    NTP configuration

    Configure the regional NTP server parameters to be applied to all machines of regional and managed clusters in the specified region.

    In the Cluster object, add the ntp:servers section with the list of required server names:

    spec:
  ...
  providerSpec:
    value:
      kaas:
      ...
      ntpEnabled: true
        regional:
          - helmReleases:
            - name: <providerName>-provider
              values:
                config:
                  lcm:
                    ...
                    ntp:
                      servers:
                      - 0.pool.ntp.org
                      ...
            provider: <providerName>
            ...

    To disable NTP:

    spec:
  ...
  providerSpec:
    value:
      ...
      ntpEnabled: false
      ...
  • audit Since 2.24.0 as TechPreview

    Optional. Auditing tools enabled on the cluster. Contains the auditd field that enables the Linux Audit daemon auditd to monitor activity of cluster processes and prevent potential malicious activity.

    Configuration for auditd

    In the Cluster object, add the auditd parameters:

    spec:
  providerSpec:
    value:
      audit:
        auditd:
          enabled: <bool>
          enabledAtBoot: <bool>
          backlogLimit: <int>
          maxLogFile: <int>
          maxLogFileAction: <string>
          maxLogFileKeep: <int>
          mayHaltSystem: <bool>
          presetRules: <string>
          customRules: <string>
          customRulesX32: <text>
          customRulesX64: <text>

    Configuration parameters for auditd:

    enabled

    Boolean, default - false. Enables the auditd role to install the auditd packages and configure rules. CIS rules: 4.1.1.1, 4.1.1.2.

    enabledAtBoot

    Boolean, default - false. Configures grub to audit processes that can be audited even if they start up prior to auditd startup. CIS rule: 4.1.1.3.

    backlogLimit

    Integer, default - none. Configures the backlog to hold records. If during boot audit=1 is configured, the backlog holds 64 records. If more than 64 records are created during boot, auditd records will be lost with a potential malicious activity being undetected. CIS rule: 4.1.1.4.

    maxLogFile

    Integer, default - none. Configures the maximum size of the audit log file. Once the log reaches the maximum size, it is rotated and a new log file is created. CIS rule: 4.1.2.1.

    maxLogFileAction

    String, default - none. Defines handling of the audit log file reaching the maximum file size. Allowed values:

    • keep_logs - rotate logs but never delete them

    • rotate - add a cron job to compress rotated log files and keep maximum 5 compressed files.

    • compress - compress log files and keep them under the /var/log/auditd/ directory. Requires auditd_max_log_file_keep to be enabled.

    CIS rule: 4.1.2.2.

    maxLogFileKeep

    Integer, default - 5. Defines the number of compressed log files to keep under the /var/log/auditd/ directory. Requires auditd_max_log_file_action=compress. CIS rules - none.

    mayHaltSystem

    Boolean, default - false. Halts the system when the audit logs are full. Applies the following configuration:

    • space_left_action = email

    • action_mail_acct = root

    • admin_space_left_action = halt

    CIS rule: 4.1.2.3.

    customRules

    String, default - none. Base64-encoded content of the 60-custom.rules file for any architecture. CIS rules - none.

    customRulesX32

    String, default - none. Base64-encoded content of the 60-custom.rules file for the i386 architecture. CIS rules - none.

    customRulesX64

    String, default - none. Base64-encoded content of the 60-custom.rules file for the x86_64 architecture. CIS rules - none.

    presetRules

    String, default - none. Comma-separated list of the following built-in preset rules:

    • access

    • actions

    • delete

    • docker

    • identity

    • immutable

    • logins

    • mac-policy

    • modules

    • mounts

    • perm-mod

    • privileged

    • scope

    • session

    • system-locale

    • time-change

    You can use two keywords for these rules:

    • none - disables all built-in rules.

    • all - enables all built-in rules. With this key, you can add the ! prefix to a rule name to exclude some rules. You can use the ! prefix for rules only if you add the all keyword as the first rule. Place a rule with the ! prefix only after the all keyword.

    Example configurations:

    • presetRules: none - disable all preset rules

    • presetRules: docker - enable only the docker rules

    • presetRules: access,actions,logins - enable only the access, actions, and logins rules

    • presetRules: all - enable all preset rules

    • presetRules: all,!immutable,!sessions - enable all preset rules except immutable and sessions


    CIS controls
    4.1.3 (time-change)
    4.1.4 (identity)
    4.1.5 (system-locale)
    4.1.6 (mac-policy)
    4.1.7 (logins)
    4.1.8 (session)
    4.1.9 (perm-mod)
    4.1.10 (access)
    4.1.11 (privileged)
    4.1.12 (mounts)
    4.1.13 (delete)
    4.1.14 (scope)
    4.1.15 (actions)
    4.1.16 (modules)
    4.1.17 (immutable)
    Docker CIS controls
    1.1.4
    1.1.8
    1.1.10
    1.1.12
    1.1.13
    1.1.15
    1.1.16
    1.1.17
    1.1.18
    1.2.3
    1.2.4
    1.2.5
    1.2.6
    1.2.7
    1.2.10
    1.2.11

    See also

    Operations Guide: Troubleshooting - The auditd events cause ‘backlog limit exceeded’ messages

  • volumesCleanupEnabled

    Available since Container Cloud 2.23.0 as Technology Preview. Schedules the volumes created using Persistent Volume Claims to be deleted during cluster deletion. Boolean, false by default.

    Caution

    The feature applies only to volumes created on clusters that are based on or updated to the Cluster release 11.7.0 or later.

    If you added volumes to an existing cluster before it was updated to the Cluster release 11.7.0, delete such volumes manually after the cluster deletion.

  • serviceAnnotations Since 2.24.0 as TechPreview

    Enables passing of any custom settings to load balancers created by Container Cloud. These annotations are set on service objects created by Container Cloud and are propagated to the OpenStack cloud provider that applies new settings to a load balancer during initial cluster deployment and creation of a new load balancer.

    Caution

    Mirantis does not recommend enabling this feature after the cluster deployment because it will apply to newly created load balancers only.

    The loadbalancer.openstack.org/flavor-id: <octaviaFlavorID> field enables creation of load balancers with defined Octavia flavors.

    For details, see OpenStack documentation: Octavia Flavors.

    Note

    This feature is not supported by OpenStack Queens.

Configuration example:

spec:
  ...
  providerSpec:
    value:
      credentials: cloud-config
      publicKeys:
        - name: demo-key
      release: release: mke-11-7-0-3-5-7
      helmReleases:
        - name: stacklight
          values:
            ...
      proxy: proxy-object-name
      tls:
        keycloak:
          tlsConfigRef: keycloak
        ui:
          tlsConfigRef: ui
      containerRegistries:
      - demoregistry
      volumesCleanupEnabled: false

spec:providerSpec for OpenStack network

The providerSpec object field of a Cluster resource contains the following fields to configure the OpenStack network:

  • dnsNameservers

    List of nameservers for the OpenStack subnet to be created

  • externalNetworkId

    ID of an external OpenStack network

  • nodeCidr

    CIDR of the OpenStack subnet to be created

Configuration example:

spec:
  ...
  providerSpec:
    value:
      dnsNameservers:
        - 172.18.224.4
      externalNetworkId: c3799996-dc8e-4477-a309-09ea6dd71946
      nodeCidr: 10.10.10.0/24

spec:providerSpec for Container Cloud configuration

This section represents the Container Cloud components that are enabled on the cluster. It contains the kaas section with the following fields:

  • management

    Configuration for the management cluster components:

    • enabled

      Cluster type:

      • true - management cluster

      • false - regional or managed cluster

    • helmReleases

      List of management cluster Helm releases that will be installed on a cluster. A Helm release includes the name and values fields. Specified values will be merged with relevant management cluster Helm release values in the Release object.

  • regional

    List of regional cluster components for each configured provider available for a specific region:

    • provider

      Provider type openstack

    • helmReleases

      List of regional Helm releases to be installed. A Helm release includes such fields as name and values. Specified values will be merged with relevant regional Helm release values in the Release object.

  • release

    Name of the Container Cloud Release object.

Configuration example:

spec:
  ...
  providerSpec:
     value:
       kaas:
         management:
           enabled: true
           helmReleases:
             - name: kaas-ui
               values:
                 serviceConfig:
                   server: <service_config>
         regional:
           - helmReleases:
             - name: <provider_name>-provider
               values: {}
             provider: <provider_name>
           - helmReleases:
             - name: <provider_name>-provider
               values: {}
             provider: <provider_name>
         release: kaas-2-0-0

spec:providerSpec for Bastion

The providerSpec object field of the Cluster resource for the Bastion instance contains the following fields:

  • image

    Image name to use for the Bastion instance.

  • redeployAllowed

    Flag that allows redeploying the Bastion host to update the SSH keys. Defaults to false.

  • flavor

    Name of the flavor to use for building the Bastion host.

  • availabilityZone

    Name of the availability zone to place the Bastion host in.

  • bootFromVolume TechPreview

    Configuration to boot the Bastion node from a block storage volume based on a given image. Required parameters:

    • enabled

      Set to true to boot from a volume.

    • volumeSize

      Size of the volume to create in GB. The default amount of storage 80 is enough for the Bastion node.

Configuration example:

spec:
  ...
  providerSpec:
    value:
      bastion:
        redeployAllowed: false
        image: bionic-server-cloudimg-amd64-20200724
        availabilityZone: nova
        flavor: kaas.small
        bootFromVolume:
           enabled: true
           volumeSize: 80

status:providerStatus

Warning

Do not modify this section using API.

The common providerStatus object field of the Cluster resource contains the following fields:

  • apiVersion

    openstackproviderconfig.k8s.io/v1alpha1

  • kind

    Object type that is OpenstackClusterProviderStatus

Configuration example:

spec:
  ...
  providerStatus:
    value:
      apiVersion: openstackproviderconfig.k8s.io/v1alpha1
      kind:  OpenstackClusterProviderStatus

status:providerStatus common

The common providerStatus object field of the Cluster resource contains the following fields:

  • loadBalancerHost

    Load balancer IP or host name of the cluster

  • loadBalancerStatus

    Load balancer status

    • id

      ID of the load balancer

    • ready

      Readiness flag

    • status

      Status details

  • apiServerCertificate

    Server certificate of Kubernetes API

  • ucpDashboard

    MKE Dashboard URL

  • maintenance

    Maintenance mode of a cluster. Prepares a cluster for maintenance and enables the possibility to switch machines into maintenance mode.

Configuration example:

status:
  ...
  providerStatus:
    loadBalancerHost: 172.16.123.456
    apiServerCertificate: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS…
    ucpDashboard: https://172.16.123.456:6443
    loadBalancerStatus:
      id: 7851a962-1deb-11eb-8bec-0242ac11
      ready: true
      status: active

status:providerStatus for OpenStack network

Warning

Do not modify this section using API.

The network section of the providerStatus object field of the Cluster resource represents basic information about the associated OpenStack network of the OpenStack cloud provider. It contains the following fields:

  • id

    Network ID

  • name

    Network name

  • loadbalancer

    Load balancer information:

    • floatingIP - floating IP of a load balancer

    • id - ID of a load balancer

    • listeners - listeners configuration of a load balancer

    • name - load balancer name

    • pools - load balancer pools IDs

  • router

    Network router information:

    • id - ID of a router

    • name - name of a router

  • subnet

    Network subnet details:

    • cidr - CIDR block of a subnet

    • id - ID of a subnet

    • name - name of a subnet

Configuration example:

status:
  providerStatus:
    network:
      id: 01234456-7890-abcd-efgh-876543219876
      name: kaas-net-abcdefgh-0123-4567-890a-0a1b2c3d4e5f
      loadbalancer:
        floatingIP: 172.19.116.5
        id: 43bc4b9d-cb44-42ed-908d-3a08dc494f5a
        listeners:
          kube-api:
            id: 8ec3dc99-ab73-401f-8036-1d3635833f0e
            port: 443
          ucp-api:
            id: f67ec325-933c-417c-af72-bfc2d7e084d6
            port: 6443
         name: kaas-lb-fc0278f0-ce30-11e9-b326-fa163e1b05fd
         pools:
           kube-api:
             id: e948ae06-3017-4345-8d63-010c5c0f57b4
           ucp-api:
             id: 4a893e8a-e915-441a-a74e-db44b1f8f8d4
      router:
        id: 7d892f20-5f6a-44ce-badc-8b040b1bfb51
        name: kaas-router-fc0278f0-ce30-11e9-b326-fa163e1b05fd
      subnet:
        cidr: 10.10.10.0/24
        id: 9515ee68-5039-476a-b35a-1c690c58a050
        name: kaas-subnet-fc0278f0-ce30-11e9-b326-fa163e1b05fd

status:providerStatus for OpenStack security groups

Warning

Do not modify this section using API.

The providerStatus object field of the Cluster resource for the OpenStack security groups contains the following fields:

  • controlPlaneSecurityGroup

    Contains the following information about the OpenStack security group rules for the control plane nodes of a Container Cloud cluster:

    • id - ID of a security group

    • name - name of a security group

    • rules - list of the security group rules

  • globalSecurityGroup

    Contains the following information about the OpenStack security group rules for all nodes of a Container Cloud cluster:

    • id - ID of a security group

    • name - name of a security group

    • rules - list of the security group rules

Configuration example:

status:
  providerStatus:
    controlPlaneSecurityGroup:
      id: 01234456-7890-abcd-efgh-876543219876
      name: kaas-sg-ctrl-abcdefgh-0123-4567-890a-0a1b2c3d4e5
      rules:
      ...
    globalSecurityGroup:
      id: 01234456-7890-abcd-efgh-876543219876
      name: kaas-sg-glob-abcdefgh-0123-4567-890a-0a1b2c3d4e5
      rules:
      ...

status:providerStatus for Bastion

The providerStatus object field of the Cluster resource for the Bastion instance contains the following fields:

  • bastion

    • publicIP - public IP of the Bastion instance

    • lcmManaged - flag indicating that Bastion is managed by LCM

  • bastionSecurityGroup

    Contains the following information about the OpenStack security group rules for the Bastion instance of a Container Cloud cluster:

    • id - ID of a security group

    • name - name of a security group

    • rules - list of the security group rules

Configuration example:

status:
 providerStatus:
   bastion:
     publicIP: 172.16.247.162
   bastionSecurityGroup:
     id: 01234456-7890-abcd-efgh-876543219876
     name: kaas-sg-bastion-abcdefgh-0123-4567-890a-0a1b2c3d4e5
     rules:
     ...

status:providerStatus for Cluster readiness

Warning

Do not modify this section using API.

The providerStatus object field of the Cluster resource that reflects cluster readiness contains the following fields:

  • persistentVolumesProviderProvisioned

    Provision status of the provider persistent volumes (PVs). Used to prevent Helm releases that require PVs from being installed until some default StorageClass is present in the cluster.

  • helm

    Status of deployed Helm releases:

    • ready

      If all Helm releases have been deployed successfully, the value switches to true.

    • releases

      List of enabled Helm Releases that run on a cluster:

      • releaseStatuses

        List of Helm releases being deployed. Each release has the success field that switches to true once a release is deployed.

      • stacklight

        Status of the StackLight deployment. Contains URLs of all StackLight components.

      • iam

        Status of the IAM deployment. Contains URLs of the keycloak and api components.

      • decc

        Status of the remaining container cloud components. Contains URLs of the ui, cache, and proxy components.

  • nodes
    • ready

      Number of nodes that completed deployment or update.

    • requested

      Total number of nodes. If the number of ready nodes does not match the number of requested nodes, it means that a cluster is being updated.

  • ceph
    • ready

      Ceph readiness flag.

    • message

      Ceph status details.

  • ready

    Cluster readiness flag. If true, the cluster is deployed successfully and all components are up and running.

  • conditions

    List of objects status condition:

    • type

      Object type

    • ready

      Readiness flag

    • message

      Status details

  • notReadyObjects

    List of Kubernetes objects (Service, Deployment, and StatefulSet) that are not in the Ready state yet:

    • Service is not ready if its external address has not been provisioned yet.

    • Deployment or StatefulSet is not ready if the number of ready replicas is not equal to the number of required replicas.

    Contains the name and namespace of the object and the number of ready and required replicas for controllers. If all objects are ready, the notReadyObjects list is empty.

Configuration example:

status:
  providerStatus:
    persistentVolumesProviderProvisioned: true
    helm:
      ready: true
      releases:
        decc:
          cache:
            url: >-
              https://a618e3d36d7f44f2e8d56bbcc53ffbf7-1765661812.us-east-2.elb.amazonaws.com
          proxy:
            url: >-
              http://a0d8d8966e0d24f50aead0942da92456-2114585625.us-east-2.elb.amazonaws.com:3128
          ui:
            url: >-
              https://a43fe72c644de41ae9db3cc77dd992d5-566275388.us-east-2.elb.amazonaws.com
        iam:
          api:
            url: >-
              https://a08d8bdd8553b49a88ab8e663d384001-1745154108.us-east-2.elb.amazonaws.com
          keycloak:
            url: >-
              https://a2b58b6a3ee3c4884b034fd791ebff6d-1687192379.us-east-2.elb.amazonaws.com
        releaseStatuses:
          admission-controller:
            success: true
          iam:
            success: true
          iam-controller:
            success: true
          kaas-exporter:
            success: true
          kaas-public-api:
            success: true
          kaas-ui:
            success: true
          lcm-controller:
          ...
        stacklight:
          alerta:
            url: http://172.16.248.170
          alertmanager:
            url: http://172.16.247.217
          grafana:
            url: http://172.16.248.49
          kibana:
            url: http://172.16.245.164
          prometheus:
            url: http://172.16.249.211
          success: true
    nodes:
      ready: 3
      requested: 3
    notReadyObjects:
      services:
        - name: testservice
          namespace: default
      deployments:
        - name: <provider_name>-provider
          namespace: kaas
          replicas: 3
          readyReplicas: 2
      statefulsets: {}
    ready: false
    ceph:
      - message: Ceph cluster has been configured successfully
        ready: true
    conditions:
      - message: Helm charts are successfully installed(upgraded).
        ready: true
        type: Helm
      - message: Kubernetes objects are fully up.
        ready: true
        type: Kubernetes
      - message: All requested nodes are ready.
        ready: true
        type: Nodes

status:providerStatus for Open ID Connect

Warning

Do not modify this section using API.

The oidc section of the providerStatus object field of the Cluster resource reflects the Open ID Connect (OIDC) configuration details. It contains the required details to obtain a cluster token and contains the following fields:

  • certificate

    Base64-encoded OIDC certificate.

  • clientId

    Client ID for OIDC requests.

  • groupsClaim

    Name of an OIDC groups claim.

  • issuerUrl

    Isuer URL to get the representation of the realm.

  • ready

    OIDC status relevance. Is true if the status fits the configuration of the LCMCluster OIDC.

Configuration example:

status:
  providerStatus:
    oidc:
      certificate: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUREekNDQWZ...
      clientId: kaas
      groupsClaim: iam_roles
      issuerUrl: https://172.16.243.211/auth/realms/iam
      ready: true

status:providerStatus for Cluster releases

Warning

Do not modify this section using API.

The releaseRefs section of the providerStatus object field of the Cluster resource provides the current Cluster release version as well as the one available for upgrade. It contains the following fields:

  • current

    Details of the currently installed Cluster release:

    • lcmType

      Type of the Cluster release (mke)

    • name

      Name of the Cluster release resource

    • version

      Release version

    • unsupportedSinceKaaSVersion

      Indicates that a newer Container Cloud release exists and it does not support the current Cluster release

  • available

    List of releases available for upgrade that contains the name and version fields

Configuration example:

status:
  providerStatus:
    releaseRefs:
      available:
        - name: mke-5-15-0-3-4-0-dev
          version: 5.15.0+3.4.0-dev
      current:
        lcmType: mke
        name: mke-5-14-0-3-3-0-beta1
        version: 5.14.0+3.3.0-beta1