Cluster¶

This section describes the vSphere Cluster resource used in Mirantis Container Cloud API. The Cluster resource describes the cluster-level parameters.

For demonstration purposes, the vSphere Cluster custom resource (CR) can be split into the following major sections:

metadata
spec:providerSpec
spec:providerSpec common
spec:providerSpec for vSphere resources
spec:providerSpec for clusterNetwork
spec:providerSpec for Container Cloud configuration
status:providerStatus common
status:providerStatus for Cluster readiness
status:providerStatus for Open ID Connect
status:providerStatus for Cluster releases

Warning

The fields in this resource are available for viewing only. They are automatically generated by the vSphere cloud provider and must not be modified using the Container Cloud API.

metadata¶

The Container Cloud Cluster custom resource (CR) contains the following fields:

apiVersion
Object API version that is cluster.k8s.io/v1alpha1.
kind
Object type that is Cluster.

The metadata object field of the Cluster resource contains the following fields:

name
Cluster name that is set using the Cluster Name field of the Create Cluster wizard in the Container Cloud web UI. For a management cluster, the cluster name can be also set using cluster.yaml.template.
namespace
Namespace in which the Cluster object is created. Management clusters are created in the default namespace. The namespace of a managed cluster matches the selected Project name in the Container Cloud web UI.
labels
Key-value pairs attached to the object:
- kaas.mirantis.com/provider
  Provider type that is vsphere for the vSphere-based clusters.
- kaas.mirantis.com/region
  Region name. The default region name for a management cluster is region-one.
  
  Note
  
  The kaas.mirantis.com/region label is removed from all Container Cloud objects in 2.26.0 (Cluster releases 17.1.0 and 16.1.0). Therefore, do not add the label starting these releases. On existing clusters updated to these releases, or if manually added, this label will be ignored by Container Cloud.

Configuration example:

apiVersion: cluster.k8s.io/v1alpha1
kind: Cluster
metadata:
  name: demo
  namespace: test
  labels:
    kaas.mirantis.com/provider: vsphere

spec:providerSpec¶

The providerSpec object field of the Cluster resource contains all required details to create the cluster-level resources. It also contains fields required for LCM deployment and the Container Cloud components integration.

The providerSpec object field is custom for each cloud provider and contains the following generic fields:

apiVersion
vsphere.cluster.k8s.io/v1alpha1
kind
Object type that is VsphereClusterProviderSpec

Configuration example:

spec:
  ...
  providerSpec:
    value:
      apiVersion: vsphere.cluster.k8s.io/v1alpha1
      kind: VsphereClusterProviderSpec

spec:providerSpec common¶

The common providerSpec object field of the Cluster resource contains the following fields:

credentials
Name of the VsphereCredential object used by the cluster to connect to the provider backend
dedicatedControlPlane
Cluster control plane nodes to be tainted, defaults to true
publicKeys
List of the PublicKey resource references
- name
  Public key name
release
Name of the ClusterRelease object to install on a cluster
helmReleases
List of enabled Helm releases from the Release object that run on a cluster
proxy
Name of the Proxy object
tls
TLS configuration for endpoints of a cluster
- keycloak
  KeyCloak endpoint
  
  tlsConfigRef
  Reference to the TLSConfig object
- ui
  Web UI endpoint
  
  tlsConfigRef
  Reference to the TLSConfig object
For more details, see TLSConfig resource.
maintenance
Maintenance mode of a cluster. Prepares a cluster for maintenance and enables the possibility to switch machines into maintenance mode.
containerRegistries
List of the ContainerRegistries resources names.

ntpEnabled

NTP server mode. Boolean, enabled by default.

Since Container Cloud 2.23.0, you can optionally disable NTP to disable the management of chrony configuration by Container Cloud and use your own system for chrony management. Otherwise, configure the regional NTP server parameters to be applied to all machines of managed clusters.

Before Container Cloud 2.23.0, you can optionally configure NTP parameters if servers from the Ubuntu NTP pool (*.ubuntu.pool.ntp.org) are accessible from the node where a management cluster is being provisioned. Otherwise, this configuration is mandatory.

audit ^{Since 2.24.0 as TechPreview}
Optional. Auditing tools enabled on the cluster. Contains the auditd field that enables the Linux Audit daemon auditd to monitor activity of cluster processes and prevent potential malicious activity.
Configuration for auditd
In the Cluster object, add the auditd parameters:

spec: providerSpec: value: audit: auditd: enabled: <bool> enabledAtBoot: <bool> backlogLimit: <int> maxLogFile: <int> maxLogFileAction: <string> maxLogFileKeep: <int> mayHaltSystem: <bool> presetRules: <string> customRules: <string> customRulesX32: <text> customRulesX64: <text>

Configuration parameters for auditd:

enabled
Boolean, default - false. Enables the auditd role to install the auditd packages and configure rules. CIS rules: 4.1.1.1, 4.1.1.2.

enabledAtBoot
Boolean, default - false. Configures grub to audit processes that can be audited even if they start up prior to auditd startup. CIS rule: 4.1.1.3.

backlogLimit
Integer, default - none. Configures the backlog to hold records. If during boot audit=1 is configured, the backlog holds 64 records. If more than 64 records are created during boot, auditd records will be lost with a potential malicious activity being undetected. CIS rule: 4.1.1.4.

maxLogFile
Integer, default - none. Configures the maximum size of the audit log file. Once the log reaches the maximum size, it is rotated and a new log file is created. CIS rule: 4.1.2.1.

maxLogFileAction
String, default - none. Defines handling of the audit log file reaching the maximum file size. Allowed values:

keep_logs - rotate logs but never delete them

rotate - add a cron job to compress rotated log files and keep maximum 5 compressed files.

compress - compress log files and keep them under the /var/log/auditd/ directory. Requires auditd_max_log_file_keep to be enabled.

CIS rule: 4.1.2.2.

maxLogFileKeep
Integer, default - 5. Defines the number of compressed log files to keep under the /var/log/auditd/ directory. Requires auditd_max_log_file_action=compress. CIS rules - none.

mayHaltSystem
Boolean, default - false. Halts the system when the audit logs are full. Applies the following configuration:

space_left_action = email

action_mail_acct = root

admin_space_left_action = halt

CIS rule: 4.1.2.3.

customRules
String, default - none. Base64-encoded content of the 60-custom.rules file for any architecture. CIS rules - none.

customRulesX32
String, default - none. Base64-encoded content of the 60-custom.rules file for the i386 architecture. CIS rules - none.

customRulesX64
String, default - none. Base64-encoded content of the 60-custom.rules file for the x86_64 architecture. CIS rules - none.

presetRules
String, default - none. Comma-separated list of the following built-in preset rules:

access

actions

delete

docker

identity

immutable

logins

mac-policy

modules

mounts

perm-mod

privileged

scope

session

system-locale

time-change

You can use two keywords for these rules:

none - disables all built-in rules.

all - enables all built-in rules. With this key, you can add the ! prefix to a rule name to exclude some rules. You can use the ! prefix for rules only if you add the all keyword as the first rule. Place a rule with the ! prefix only after the all keyword.

Example configurations:

presetRules: none - disable all preset rules

presetRules: docker - enable only the docker rules

presetRules: access,actions,logins - enable only the access, actions, and logins rules

presetRules: all - enable all preset rules

presetRules: all,!immutable,!sessions - enable all preset rules except immutable and sessions

CIS controls

4.1.3 (time-change)

4.1.4 (identity)

4.1.5 (system-locale)

4.1.6 (mac-policy)

4.1.7 (logins)

4.1.8 (session)

4.1.9 (perm-mod)

4.1.10 (access)

4.1.11 (privileged)

4.1.12 (mounts)

4.1.13 (delete)

4.1.14 (scope)

4.1.15 (actions)

4.1.16 (modules)

4.1.17 (immutable)

Docker CIS controls

1.1.4

1.1.8

1.1.10

1.1.12

1.1.13

1.1.15

1.1.16

1.1.17

1.1.18

1.2.3

1.2.4

1.2.5

1.2.6

1.2.7

1.2.10

1.2.11
See also

Operations Guide: Troubleshooting - The auditd events cause ‘backlog limit exceeded’ messages

loadBalancerHost
IP of the built-in load balancer for the cluster API.

Configuration example:

spec:
  ...
  providerSpec:
    value:
      credentials: cloud-config
      publicKeys:
        - name: demo-key
      release: release: mke-5-16-0-3-3-6
      helmReleases:
        - name: stacklight
          values:
            ...
      proxy: proxy-object-name-example
      tls:
        keycloak:
          certificate:
            name: keycloak
          hostname: container-cloud-auth.example.com
        ui:
          certificate:
            name: ui
          hostname: container-cloud-ui.example.com
      containerRegistries:
      - demoregistry
      ntpEnabled: false
      ...
      loadBalancerHost: 172.16.1.21

spec:providerSpec for vSphere resources¶

The vsphere section in spec:providerSpec contains the vSphere resources configuration. For more details about vSphere resources, see Requirements for deployment resources.

The vsphere section contains the following fields:

vsphere
vSphere resources configuration:
- cloudProviderDatastore
  Datastore for Kubernetes volumes.
- clusterApiDatastore
  Datastore for cluster machines disks.
- machineFolderPath
  Folder to store cluster machines on vSphere.
- networkPath
  Path to the vSphere network.
- resourcePoolPath
  Path to the vSphere resource pool.
- scsiControllerType
  Small Computer System Interface (SCSI) controller type that is pvscsi. Other types are not supported.

Configuration example:

spec:
  ...
  providerSpec:
    value:
      apiVersion: vsphere.cluster.k8s.io/v1alpha1
      ...
      vsphere:
        cloudProviderDatastore: /DATACENTER/datastore/storage-example
        clusterApiDatastore: /DATACENTER/datastore/storage-example
        machineFolderPath: /DATACENTER/vm/vm-folder
        networkPath: /DATACENTER/network/VMWare_Network
        resourcePoolPath: /DATACENTER/host/ClusterName/Resources/ResPoolName
        scsiControllerType: pvscsi

spec:providerSpec for clusterNetwork¶

The spec:providerSpec section for clusterNetwork configuration contains the following fields:

clusterNetwork
Cluster network configuration:
- ipamEnabled
  Option to enable static IP address management. Set to true for networks without DHCP.
  
  Caution
  
  The following fields are mandatory only if IPAM is enabled. Otherwise, they do not apply.
  
  Note
  
  To obtain IPAM parameters for the selected vSphere network, contact your vSphere administrator who provides you with IP ranges dedicated to your environment only.
- cidr
  CIDR of the provided vSphere network.
- gateway
  Gateway of the provided vSphere network.
- nameservers
  List of nameservers for the network.
- includeRanges
  IP range for cluster machines. Specify the range of the provided CIDR. For example, 10.20.0.100-10.20.0.200.
- excludeRanges
  Optional. IP ranges to be excluded from being assigned to the cluster machines. The MetalLB range and loadBalancerHost should not intersect with the addresses for IPAM. For example, 10.20.0.150-10.20.0.170.

Configuration example:

spec:
  ...
  providerSpec:
    value:
      apiVersion: vsphere.cluster.k8s.io/v1alpha1
      ...
      clusterNetwork:
        cidr: 172.16.1.0/24
        gateway: 172.16.1.1
        includeRanges:
        - 172.16.1.10-172.16.1.20
        ipamEnabled: true
        nameservers:
        - 172.16.1.100
        - 172.16.1.200
        ...

spec:providerSpec for Container Cloud configuration¶

This section represents the Container Cloud components that are enabled on the cluster. It contains the kaas section with the following fields:

management
Configuration for the management cluster components:
- enabled
  Cluster type:
  
  true - management cluster
  
  false - managed cluster
- helmReleases
  List of management cluster Helm releases that will be installed on a cluster. A Helm release includes the name and values fields. Specified values will be merged with relevant management cluster Helm release values in the Release object.
regional
List of regional cluster components of the Container Cloud cluster for the configured provider:
- provider
  Provider type vsphere
- helmReleases
  List of regional Helm releases to be installed. A Helm release includes such fields as name and values. Specified values will be merged with relevant regional Helm release values in the Release object.
release
Name of the Container Cloud Release object.

Configuration example:

spec:
  ...
  providerSpec:
     value:
       kaas:
         management:
           enabled: true
           helmReleases:
             - name: kaas-ui
               values:
                 serviceConfig:
                   server: <service_config>
         regional:
           - helmReleases:
             - name: <provider_name>-provider
               values: {}
             provider: <provider_name>
         release: kaas-2-0-0

status:providerStatus common¶

The common providerStatus object field of the Cluster resource contains the following fields:

loadBalancerHost
Load balancer IP or host name of the cluster
loadBalancerStatus
Load balancer status
- id
  ID of the load balancer
- ready
  Readiness flag
- status
  Status details
apiServerCertificate
Server certificate of Kubernetes API
ucpDashboard
MKE Dashboard URL
maintenance
Maintenance mode of a cluster. Prepares a cluster for maintenance and enables the possibility to switch machines into maintenance mode.

Configuration example:

status:
  ...
  providerStatus:
    loadBalancerHost: 172.16.123.456
    apiServerCertificate: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS…
    ucpDashboard: https://172.16.123.456:6443
    loadBalancerStatus:
      id: 7851a962-1deb-11eb-8bec-0242ac11
      ready: true
      status: active

status:providerStatus for Cluster readiness¶

Warning

Do not modify this section using API.

The providerStatus object field of the Cluster resource that reflects cluster readiness contains the following fields:

persistentVolumesProviderProvisioned
Provision status of the provider persistent volumes (PVs). Used to prevent Helm releases that require PVs from being installed until some default StorageClass is present in the cluster.
helm
Status of deployed Helm releases:
- ready
  If all Helm releases have been deployed successfully, the value switches to true.
- releases
  List of enabled Helm Releases that run on a cluster:
  
  releaseStatuses
  List of Helm releases being deployed. Each release has the success field that switches to true once a release is deployed.
  
  stacklight
  Status of the StackLight deployment. Contains URLs of all StackLight components.
  
  iam
  Status of the IAM deployment. Contains URLs of the keycloak and api components.
  
  decc
  Status of the remaining container cloud components. Contains URLs of the ui, cache, and proxy components.
nodes
- ready
  Number of nodes that completed deployment or update.
- requested
  Total number of nodes. If the number of ready nodes does not match the number of requested nodes, it means that a cluster is being updated.
ceph
- ready
  Ceph readiness flag.
- message
  Ceph status details.
ready
Cluster readiness flag. If true, the cluster is deployed successfully and all components are up and running.
conditions
List of objects status condition:
- type
  Object type
- ready
  Readiness flag
- message
  Status details
notReadyObjects
List of Kubernetes objects (Service, Deployment, and StatefulSet) that are not in the Ready state yet:
- Service is not ready if its external address has not been provisioned yet.
- Deployment or StatefulSet is not ready if the number of ready replicas is not equal to the number of required replicas.
Contains the name and namespace of the object and the number of ready and required replicas for controllers. If all objects are ready, the notReadyObjects list is empty.

Configuration example:

status:
  providerStatus:
    persistentVolumesProviderProvisioned: true
    helm:
      ready: true
      releases:
        decc:
          cache:
            url: >-
              https://a618e3d36d7f44f2e8d56bbcc53ffbf7-1765661812.us-east-2.elb.amazonaws.com
          proxy:
            url: >-
              http://a0d8d8966e0d24f50aead0942da92456-2114585625.us-east-2.elb.amazonaws.com:3128
          ui:
            url: >-
              https://a43fe72c644de41ae9db3cc77dd992d5-566275388.us-east-2.elb.amazonaws.com
        iam:
          api:
            url: >-
              https://a08d8bdd8553b49a88ab8e663d384001-1745154108.us-east-2.elb.amazonaws.com
          keycloak:
            url: >-
              https://a2b58b6a3ee3c4884b034fd791ebff6d-1687192379.us-east-2.elb.amazonaws.com
        releaseStatuses:
          admission-controller:
            success: true
          iam:
            success: true
          iam-controller:
            success: true
          kaas-exporter:
            success: true
          kaas-public-api:
            success: true
          kaas-ui:
            success: true
          lcm-controller:
          ...
        stacklight:
          alerta:
            url: http://172.16.248.170
          alertmanager:
            url: http://172.16.247.217
          grafana:
            url: http://172.16.248.49
          kibana:
            url: http://172.16.245.164
          prometheus:
            url: http://172.16.249.211
          success: true
    nodes:
      ready: 3
      requested: 3
    notReadyObjects:
      services:
        - name: testservice
          namespace: default
      deployments:
        - name: <provider_name>-provider
          namespace: kaas
          replicas: 3
          readyReplicas: 2
      statefulsets: {}
    ready: false
    ceph:
      - message: Ceph cluster has been configured successfully
        ready: true
    conditions:
      - message: Helm charts are successfully installed(upgraded).
        ready: true
        type: Helm
      - message: Kubernetes objects are fully up.
        ready: true
        type: Kubernetes
      - message: All requested nodes are ready.
        ready: true
        type: Nodes

status:providerStatus for Open ID Connect¶

Warning

Do not modify this section using API.

The oidc section of the providerStatus object field of the Cluster resource reflects the Open ID Connect (OIDC) configuration details. It contains the required details to obtain a cluster token and contains the following fields:

certificate
Base64-encoded OIDC certificate.
clientId
Client ID for OIDC requests.
groupsClaim
Name of an OIDC groups claim.
issuerUrl
Isuer URL to get the representation of the realm.
ready
OIDC status relevance. Is true if the status fits the configuration of the LCMCluster OIDC.

Configuration example:

status:
  providerStatus:
    oidc:
      certificate: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUREekNDQWZ...
      clientId: kaas
      groupsClaim: iam_roles
      issuerUrl: https://172.16.243.211/auth/realms/iam
      ready: true

status:providerStatus for Cluster releases¶

Warning

Do not modify this section using API.

The releaseRefs section of the providerStatus object field of the Cluster resource provides the current Cluster release version as well as the one available for upgrade. It contains the following fields:

current
Details of the currently installed Cluster release:
- lcmType
  Type of the Cluster release (mke)
- name
  Name of the Cluster release resource
- version
  Release version
- unsupportedSinceKaaSVersion
  Indicates that a newer Container Cloud release exists and it does not support the current Cluster release
available
List of releases available for upgrade that contains the name and version fields

Configuration example:

status:
  providerStatus:
    releaseRefs:
      available:
        - name: mke-5-15-0-3-4-0-dev
          version: 5.15.0+3.4.0-dev
      current:
        lcmType: mke
        name: mke-5-14-0-3-3-0-beta1
        version: 5.14.0+3.3.0-beta1