Cluster¶

This section describes the Cluster resource used the in Mirantis Container Cloud API that describes the cluster-level parameters.

For demonstration purposes, the Container Cloud Cluster custom resource (CR) is split into the following major sections:

metadata
spec:providerSpec
spec:providerSpec common
spec:providerSpec configuration
status:providerStatus common
status:providerStatus for cluster readiness
status:providerStatus for Open ID Connect
status:providerStatus for cluster releases

Warning

The fields of the Cluster resource that are located under the status section including providerStatus are available for viewing only. They are automatically generated by the bare metal cloud provider and must not be modified using Container Cloud API.

metadata¶

The Container Cloud Cluster CR contains the following fields:

apiVersion
API version of the object that is cluster.k8s.io/v1alpha1.
kind
Object type that is Cluster.

The metadata object field of the Cluster resource contains the following fields:

name
Name of a cluster. A managed cluster name is specified under the Cluster Name field in the Create Cluster wizard of the Container Cloud web UI. A management cluster name is configurable in the bootstrap script.
namespace
Project in which the cluster object was created. The management cluster is always created in the default project. The managed cluster project equals to the selected project name.
labels
Key-value pairs attached to the object:
- kaas.mirantis.com/provider
  Provider type that is baremetal for the baremetal-based clusters.
- kaas.mirantis.com/region
  Region name. The default region name for the management cluster is region-one.
  
  Note
  
  The kaas.mirantis.com/region label is removed from all Container Cloud objects in 2.26.0 (Cluster releases 17.1.0 and 16.1.0). Therefore, do not add the label starting these releases. On existing clusters updated to these releases, or if manually added, this label will be ignored by Container Cloud.
Warning

Labels and annotations that are not documented in this API Reference are generated automatically by Container Cloud. Do not modify them using the Container Cloud API.

Configuration example:

apiVersion: cluster.k8s.io/v1alpha1
kind: Cluster
metadata:
  name: demo
  namespace: test
  labels:
    kaas.mirantis.com/provider: baremetal

spec:providerSpec¶

The spec object field of the Cluster object represents the BaremetalClusterProviderSpec subresource that contains a complete description of the desired bare metal cluster state and all details to create the cluster-level resources. It also contains the fields required for LCM deployment and integration of the Container Cloud components.

The providerSpec object field is custom for each cloud provider and contains the following generic fields for the bare metal provider:

apiVersion
API version of the object that is baremetal.k8s.io/v1alpha1
kind
Object type that is BaremetalClusterProviderSpec

Configuration example:

spec:
  ...
  providerSpec:
    value:
      apiVersion: baremetal.k8s.io/v1alpha1
      kind: BaremetalClusterProviderSpec

spec:providerSpec common¶

The common providerSpec object field of the Cluster resource contains the following fields:

credentials
Field reserved for other cloud providers, has an empty value. Disregard this field.

dedicatedControlPlane
Cluster control plane nodes to be tainted, defaults to true
publicKeys
List of the PublicKey resource references
- name
  Public key name
release
Name of the ClusterRelease object to install on a cluster
helmReleases
List of enabled Helm releases from the Release object that run on a cluster
proxy
Name of the Proxy object
tls
TLS configuration for endpoints of a cluster
- keycloak
  KeyCloak endpoint
  
  tlsConfigRef
  Reference to the TLSConfig object
- ui
  Web UI endpoint
  
  tlsConfigRef
  Reference to the TLSConfig object
For more details, see TLSConfig resource.
maintenance
Maintenance mode of a cluster. Prepares a cluster for maintenance and enables the possibility to switch machines into maintenance mode.
containerRegistries
List of the ContainerRegistries resources names.

ntpEnabled

NTP server mode. Boolean, enabled by default.

Since Container Cloud 2.23.0, you can optionally disable NTP to disable the management of chrony configuration by Container Cloud and use your own system for chrony management. Otherwise, configure the regional NTP server parameters to be applied to all machines of managed clusters.

Before Container Cloud 2.23.0, you can optionally configure NTP parameters if servers from the Ubuntu NTP pool (*.ubuntu.pool.ntp.org) are accessible from the node where a management cluster is being provisioned. Otherwise, this configuration is mandatory.

audit ^{Since 2.24.0 as TechPreview}
Optional. Auditing tools enabled on the cluster. Contains the auditd field that enables the Linux Audit daemon auditd to monitor activity of cluster processes and prevent potential malicious activity.
Configuration for auditd
In the Cluster object, add the auditd parameters:

spec: providerSpec: value: audit: auditd: enabled: <bool> enabledAtBoot: <bool> backlogLimit: <int> maxLogFile: <int> maxLogFileAction: <string> maxLogFileKeep: <int> mayHaltSystem: <bool> presetRules: <string> customRules: <string> customRulesX32: <text> customRulesX64: <text>

Configuration parameters for auditd:

enabled
Boolean, default - false. Enables the auditd role to install the auditd packages and configure rules. CIS rules: 4.1.1.1, 4.1.1.2.

enabledAtBoot
Boolean, default - false. Configures grub to audit processes that can be audited even if they start up prior to auditd startup. CIS rule: 4.1.1.3.

backlogLimit
Integer, default - none. Configures the backlog to hold records. If during boot audit=1 is configured, the backlog holds 64 records. If more than 64 records are created during boot, auditd records will be lost with a potential malicious activity being undetected. CIS rule: 4.1.1.4.

maxLogFile
Integer, default - none. Configures the maximum size of the audit log file. Once the log reaches the maximum size, it is rotated and a new log file is created. CIS rule: 4.1.2.1.

maxLogFileAction
String, default - none. Defines handling of the audit log file reaching the maximum file size. Allowed values:

keep_logs - rotate logs but never delete them

rotate - add a cron job to compress rotated log files and keep maximum 5 compressed files.

compress - compress log files and keep them under the /var/log/auditd/ directory. Requires auditd_max_log_file_keep to be enabled.

CIS rule: 4.1.2.2.

maxLogFileKeep
Integer, default - 5. Defines the number of compressed log files to keep under the /var/log/auditd/ directory. Requires auditd_max_log_file_action=compress. CIS rules - none.

mayHaltSystem
Boolean, default - false. Halts the system when the audit logs are full. Applies the following configuration:

space_left_action = email

action_mail_acct = root

admin_space_left_action = halt

CIS rule: 4.1.2.3.

customRules
String, default - none. Base64-encoded content of the 60-custom.rules file for any architecture. CIS rules - none.

customRulesX32
String, default - none. Base64-encoded content of the 60-custom.rules file for the i386 architecture. CIS rules - none.

customRulesX64
String, default - none. Base64-encoded content of the 60-custom.rules file for the x86_64 architecture. CIS rules - none.

presetRules
String, default - none. Comma-separated list of the following built-in preset rules:

access

actions

delete

docker

identity

immutable

logins

mac-policy

modules

mounts

perm-mod

privileged

scope

session

system-locale

time-change

You can use two keywords for these rules:

none - disables all built-in rules.

all - enables all built-in rules. With this key, you can add the ! prefix to a rule name to exclude some rules. You can use the ! prefix for rules only if you add the all keyword as the first rule. Place a rule with the ! prefix only after the all keyword.

Example configurations:

presetRules: none - disable all preset rules

presetRules: docker - enable only the docker rules

presetRules: access,actions,logins - enable only the access, actions, and logins rules

presetRules: all - enable all preset rules

presetRules: all,!immutable,!sessions - enable all preset rules except immutable and sessions

CIS controls

4.1.3 (time-change)

4.1.4 (identity)

4.1.5 (system-locale)

4.1.6 (mac-policy)

4.1.7 (logins)

4.1.8 (session)

4.1.9 (perm-mod)

4.1.10 (access)

4.1.11 (privileged)

4.1.12 (mounts)

4.1.13 (delete)

4.1.14 (scope)

4.1.15 (actions)

4.1.16 (modules)

4.1.17 (immutable)

Docker CIS controls

1.1.4

1.1.8

1.1.10

1.1.12

1.1.13

1.1.15

1.1.16

1.1.17

1.1.18

1.2.3

1.2.4

1.2.5

1.2.6

1.2.7

1.2.10

1.2.11
See also

Operations Guide: Troubleshooting - The auditd events cause ‘backlog limit exceeded’ messages
secureOverlay ^{TechPreview since 2.24.0 and 2.24.2 for MOSK 23.2}
Optional. Enables WireGuard for traffic encryption on the Kubernetes workloads network. Boolean. Disabled by default.

Caution

Before enabling WireGuard, ensure that the Calico MTU size is at least 60 bytes smaller than the interface MTU size of the workload network. IPv4 WireGuard uses a 60-byte header. For details, see Set the MTU size for Calico.

Caution

Changing this parameter on a running cluster causes a downtime that can vary depending on the cluster size.

For more details about WireGuard, see Calico documentation: Encrypt in-cluster pod traffic.

useBGPAnnouncement ^{Since 2.24.4 as TechPreview}
Optional. To enable the use of BGP announcement for the cluster API LB address, set to true. See Configure BGP announcement for cluster API LB address for details.

Configuration example:

spec:
  ...
  providerSpec:
    value:
      credentials: ""
      publicKeys:
        - name: bootstrap-key
      release: ucp-5-7-0-3-3-3-tp11
      helmReleases:
        - name: metallb
          values:
            configInline:
              address-pools:
                - addresses:
                  - 10.0.0.101-10.0.0.120
                    name: default
                    protocol: layer2
        ...
        - name: stacklight
          ...
      tls:
        keycloak:
          certificate:
            name: keycloak
          hostname: container-cloud-auth.example.com
        ui:
          certificate:
            name: ui
          hostname: container-cloud-ui.example.com
      containerRegistries:
      - demoregistry
      ntpEnabled: false
      ...

spec:providerSpec configuration¶

This section represents the Container Cloud components that are enabled on a cluster. It contains the following fields:

management
Configuration for the management cluster components:
- enabled
  Management cluster enabled (true) or disabled (false).
- helmReleases
  List of the management cluster Helm releases that will be installed on the cluster. A Helm release includes the name and values fields. The specified values will be merged with relevant Helm release values of the management cluster in the Release object.
regional
List of regional cluster components for the provider:
- provider
  Provider type that is baremetal.
- helmReleases
  List of the regional Helm releases that will be installed on the cluster. A Helm release includes the name and values fields. The specified values will be merged with relevant regional Helm release values in the Release object.
release
Name of the Container Cloud Release object.

Configuration example:

spec:
  ...
  providerSpec:
     value:
       kaas:
         management:
           enabled: true
           helmReleases:
             - name: kaas-ui
               values:
                 serviceConfig:
                   server: https://10.0.0.117
         regional:
           - helmReleases:
             - name: baremetal-provider
               values: {}
             provider: baremetal
           ...
         release: kaas-2-0-0

status:providerStatus common¶

^{Must not be modified using API}

The common providerStatus object field of the Cluster resource contains the following fields:

apiVersion
API version of the object that is baremetal.k8s.io/v1alpha1
kind
Object type that is BaremetalClusterProviderStatus
loadBalancerHost
Load balancer IP or host name of the Container Cloud cluster
apiServerCertificate
Server certificate of Kubernetes API
ucpDashboard
URL of the Mirantis Kubernetes Engine (MKE) Dashboard
maintenance
Maintenance mode of a cluster. Prepares a cluster for maintenance and enables the possibility to switch machines into maintenance mode.

Configuration example:

status:
  providerStatus:
    apiVersion: baremetal.k8s.io/v1alpha1
    kind: BaremetalClusterProviderStatus
    loadBalancerHost: 10.0.0.100
    apiServerCertificate: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS…
    ucpDashboard: https://10.0.0.100:6443

status:providerStatus for cluster readiness¶

^{Must not be modified using API}

The providerStatus object field of the Cluster resource that reflects the cluster readiness contains the following fields:

persistentVolumesProviderProvisioned
Status of the persistent volumes provisioning. Prevents the Helm releases that require persistent volumes from being installed until some default StorageClass is added to the Cluster object.
helm
Details about the deployed Helm releases:
- ready
  Status of the deployed Helm releases. The true value indicates that all Helm releases are deployed successfully.
- releases
  List of the enabled Helm releases that run on the Container Cloud cluster:
  
  releaseStatuses
  List of the deployed Helm releases. The success: true field indicates that the release is deployed successfully.
  
  stacklight
  Status of the StackLight deployment. Contains URLs of all StackLight components. The success: true field indicates that StackLight is deployed successfully.
nodes
Details about the cluster nodes:
- ready
  Number of nodes that completed the deployment or update.
- requested
  Total number of nodes. If the number of ready nodes does not match the number of requested nodes, it means that a cluster is being currently deployed or updated.
notReadyObjects
The list of the services, deployments, and statefulsets Kubernetes objects that are not in the Ready state yet. A service is not ready if its external address has not been provisioned yet. A deployment or statefulset is not ready if the number of ready replicas is not equal to the number of desired replicas. Both objects contain the name and namespace of the object and the number of ready and desired replicas (for controllers). If all objects are ready, the notReadyObjects list is empty.

Configuration example:

status:
  providerStatus:
    persistentVolumesProviderProvisioned: true
    helm:
      ready: true
      releases:
        releaseStatuses:
          iam:
            success: true
          ...
        stacklight:
          alerta:
            url: http://10.0.0.106
          alertmanager:
            url: http://10.0.0.107
          grafana:
            url: http://10.0.0.108
          kibana:
            url: http://10.0.0.109
          prometheus:
            url: http://10.0.0.110
          success: true
    nodes:
      ready: 3
      requested: 3
    notReadyObjects:
      services:
        - name: testservice
          namespace: default
      deployments:
        - name: baremetal-provider
          namespace: kaas
          replicas: 3
          readyReplicas: 2
      statefulsets: {}

status:providerStatus for Open ID Connect¶

^{Must not be modified using API}

The oidc section of the providerStatus object field in the Cluster resource reflects the Open ID Connect configuration details. It contains the required details to obtain a token for a Container Cloud cluster and consists of the following fields:

certificate
Base64-encoded OIDC certificate.
clientId
Client ID for OIDC requests.
groupsClaim
Name of an OIDC groups claim.
issuerUrl
Issuer URL to obtain the representation of the realm.
ready
OIDC status relevance. If true, the status corresponds to the LCMCluster OIDC configuration.

Configuration example:

status:
  providerStatus:
    oidc:
      certificate: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUREekNDQWZ...
      clientId: kaas
      groupsClaim: iam_roles
      issuerUrl: https://10.0.0.117/auth/realms/iam
      ready: true

status:providerStatus for cluster releases¶

^{Must not be modified using API}

The releaseRefs section of the providerStatus object field in the Cluster resource provides the current Cluster release version as well as the one available for upgrade. It contains the following fields:

current
Details of the currently installed Cluster release:
- lcmType
  Type of the Cluster release (ucp).
- name
  Name of the Cluster release resource.
- version
  Version of the Cluster release.
- unsupportedSinceKaaSVersion
  Indicates that a Container Cloud release newer than the current one exists and that it does not support the current Cluster release.
available
List of the releases available for upgrade. Contains the name and version fields.

Configuration example:

status:
  providerStatus:
    releaseRefs:
      available:
        - name: ucp-5-5-0-3-4-0-dev
          version: 5.5.0+3.4.0-dev
      current:
        lcmType: ucp
        name: ucp-5-4-0-3-3-0-beta1
        version: 5.4.0+3.3.0-beta1