Cluster¶

This section describes the Cluster resource used in Mirantis Container Cloud API. The Cluster resource describes the cluster-level parameters.

For demonstration purposes, the Container Cloud Cluster custom resource (CR) can be split into the following major sections:

metadata
spec:providerSpec
spec:providerSpec common
spec:providerSpec:networkSpec for AWS network
spec:providerSpec for Container Cloud configuration
spec:providerSpec for Bastion
status:providerStatus common
status:providerStatus for AWS network
status:providerStatus for Bastion
status:providerStatus for Cluster readiness
status:providerStatus for Open ID Connect
status:providerStatus for Cluster releases

Warning

The fields of the Cluster resource that are located under the status section including providerStatus are available for viewing only. They are automatically generated by the AWS cloud provider and must not be modified using Container Cloud API.

metadata¶

The Container Cloud Cluster custom resource (CR) contains the following fields:

apiVersion
cluster.k8s.io/v1alpha1
kind
Object type that is Cluster

The metadata object field of the Cluster resource contains the following fields:

name
Cluster name. A managed cluster name is set in the Cluster Name field of the Create Cluster wizard in the Container Cloud web UI. Management and regional cluster names are set in the bootstrap script.
namespace
Namespace in which the Cluster object is created. Management and regional clusters are created in the default namespace. The namespace of a managed cluster matches the selected Project name in the Container Cloud web UI.
labels
Key-value pairs attached to the object:
- kaas.mirantis.com/provider
  Provider type that is aws for the AWS-based clusters
- kaas.mirantis.com/region
  Region name. The default region name for a management cluster is region-one. For regional clusters, it is configurable using the REGION parameter in the bootstrap script.
Warning

Labels and annotations that are not documented in this API Reference are generated automatically by Container Cloud. Do not modify them using the Container Cloud API.

Configuration example:

apiVersion: cluster.k8s.io/v1alpha1
kind: Cluster
metadata:
  name: demo
  namespace: test
  labels:
    kaas.mirantis.com/provider: aws
    kaas.mirantis.com/region: region-one

spec:providerSpec¶

The providerSpec object field of the Cluster resource contains all required details to create the cluster-level resources. It also contains fields required for LCM deployment and the Container Cloud components integration.

The providerSpec object field is custom for each cloud provider and contains the following generic fields:

apiVersion
aws.kaas.mirantis.com/v1alpha1
kind
Object type that is AWSClusterProviderSpec

Configuration example:

spec:
  ...
  providerSpec:
    value:
      apiVersion: aws.kaas.mirantis.com/v1alpha1
      kind: AWSClusterProviderSpec

spec:providerSpec common¶

The common providerSpec object field of the Cluster resource contains the following fields:

credentials
Name of the AWSCredential object used by the cluster to connect to the provider back end
dedicatedControlPlane
Cluster control plane nodes to be tainted, defaults to true
publicKeys
List of the PublicKey resource references
- name
  Public key name
release
Name of the ClusterRelease object to install on a cluster
helmReleases
List of enabled Helm releases from the Release object that run on a cluster
proxy
Name of the Proxy object
tls
TLS configuration for endpoints of a cluster
- keycloak
  KeyCloak endpoint
  
  tlsConfigRef
  Reference to the TLSConfig object
- ui
  Web UI endpoint
  
  tlsConfigRef
  Reference to the TLSConfig object
For more details, see TLSConfig resource.
maintenance
Maintenance mode of a cluster. Prepares a cluster for maintenance and enables the possibility to switch machines into maintenance mode.
containerRegistries
List of the ContainerRegistries resources names.

Note

For MOSK-based deployments, the feature support is available since MOSK 22.3.

region
AWS region where the cluster is located

Configuration example:

spec:
  ...
  providerSpec:
    value:
      credentials: cloud-config
      publicKeys:
        - name: demo-key
      release: mke-5-16-0-3-3-6
      helmReleases:
        - name: stacklight
          values:
            ...
      region: us-east-2
      tls:
        keycloak:
          certificate:
            name: keycloak
          hostname: container-cloud-auth.example.com
        ui:
          certificate:
            name: ui
          hostname: container-cloud-ui.example.com

spec:providerSpec:networkSpec for AWS network¶

The networkSpec object field of the Cluster resource contains the following fields to configure the AWS Network:

subnets
Subnets configuration that contains the following fields:
- availabilityZone
  Availability zone to use for this subnet in the cluster region.
- cidrBlock
  CIDR block to be used when the provider creates a managed VPC.
- id
  Unique identifier to reference the subnet.
- isPublic
  If true, a subnet is public. A subnet is public when it is associated with a route table that has a route to an Internet gateway.
- natGatewayId
  NAT gateway ID associated with the subnet.
- routeTableId
  Routing table ID associated with the subnet.
- tags
  List of tags describing the resource. Contains the key-value pairs that describe the subnet name, cluster UID, subnet role, and so on.
vpc
VPC configuration:
- cidrBlock
  CIDR block to be used when the provider creates a managed VPC. Defaults to 10.0.0.0/16.
- id
  ID of the VPC to be used when the provider creates resources
- internetGatewayId
  ID of the Internet gateway associated with the VPC
- tags
  List of tags describing the resource. It contains the key-value pairs that describe the VPC name, cluster UID, VPC role, and so on.

Configuration example:

spec:
  ...
  providerSpec:
    value:
      networkSpec:
        subnets:
          - availabilityZone: us-east-2a
            cidrBlock: 10.0.0.0/24
            id: subnet-04cddf5343d03b32b
            isPublic: true
            natGatewayId: nat-039a062cb85ca45ce
            routeTableId: rtb-05da80001e5c426aa
            tags:
              Name: 8dcf6cc2-f350-11ea-875a-0242ac110002-subnet-public
              kubernetes.io/cluster/8dcf6cc2-f350-XXX-0242ac110002: owned
              sigs.k8s.io/cluster-api-provider-aws/managed: "true"
              sigs.k8s.io/cluster-api-provider-aws/role: public
            ...
        vpc:
          cidrBlock: 10.0.0.0/16
          id: vpc-04e008da653bdb05f
          internetGatewayId: igw-0e2164ed8de6c8d2b
          tags:
            Name: 8dcf6cc2-f350-11ea-875a-0242ac110002-vpc
            kubernetes.io/cluster/8dcf6cc2-f350-XXX-0242ac110002: owned
            sigs.k8s.io/cluster-api-provider-aws/managed: "true"
            sigs.k8s.io/cluster-api-provider-aws/role: common

spec:providerSpec for Container Cloud configuration¶

This section represents the Container Cloud components that are enabled on the cluster. It contains the kaas section with the following fields:

management
Configuration for the management cluster components:
- enabled
  Cluster type:
  
  true - management cluster
  
  false - regional or managed cluster
- helmReleases
  List of management cluster Helm releases that will be installed on a cluster. A Helm release includes the name and values fields. Specified values will be merged with relevant management cluster Helm release values in the Release object.
regional
List of regional cluster components for each configured provider available for a specific region:
- provider
  Provider type aws
- helmReleases
  List of regional Helm releases to be installed. A Helm release includes such fields as name and values. Specified values will be merged with relevant regional Helm release values in the Release object.
release
Name of the Container Cloud Release object.

Configuration example:

spec:
  ...
  providerSpec:
     value:
       kaas:
         management:
           enabled: true
           helmReleases:
             - name: kaas-ui
               values:
                 serviceConfig:
                   server: <service_config>
         regional:
           - helmReleases:
             - name: <provider_name>-provider
               values: {}
             provider: <provider_name>
           - helmReleases:
             - name: byo-provider
               values: {}
             provider: byo
         release: kaas-2-0-0

spec:providerSpec for Bastion¶

The providerSpec object field of the Cluster resource for the Bastion instance contains the following fields:

image
Image name to use for the Bastion instance.
redeployAllowed
Flag that allows redeploying the Bastion host to update the SSH keys. Defaults to false.
amiId
The ID of the AMI used to launch the bastion instance.

Configuration example:

providerSpec:
  value:
    bastion:
      redeployAllowed: false
      amiId: ami-2581aa40

status:providerStatus common¶

The common providerStatus object field of the Cluster resource contains the following fields:

loadBalancerHost
Load balancer IP or host name of the cluster
loadBalancerStatus
Load balancer status
- id
  ID of the load balancer
- ready
  Readiness flag
- status
  Status details
apiServerCertificate
Server certificate of Kubernetes API
ucpDashboard
MKE Dashboard URL
maintenance
Maintenance mode of a cluster. Prepares a cluster for maintenance and enables the possibility to switch machines into maintenance mode.

Configuration example:

status:
  ...
  providerStatus:
    loadBalancerHost: 172.16.123.456
    apiServerCertificate: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS…
    ucpDashboard: https://172.16.123.456:6443
    loadBalancerStatus:
      id: 7851a962-1deb-11eb-8bec-0242ac11
      ready: true
      status: active

status:providerStatus for AWS network¶

Warning

Do not modify this section using API.

The network section of the providerStatus object field of the Cluster resource represents basic information about the associated AWS network. It contains the following fields:

apiServerNetworkElb
The Kubernetes API server network load balancer:
- arn
  ARN to uniquely identify AWS resources.
- attributes
  List of extra attributes associated with the load balancer.
- dnsName
  DNS name of the load balancer.
- name
  Name of the load balancer.
- scheme
  Scheme of the load balancer that is Internet-facing or private.
securityGroups
Security groups configuration for the Bastion instance (bastion) and cluster nodes (controlplane and node). Each subfield contains the following security group configuration:
- id
  Unique identifier of the security group.
- ingressRule
  Inbound rules associated with the security group.
- name
  Security group name.
- tags
  List of key-value pairs of tags associated with the security group.

Configuration example:

status:
  providerStatus:
    network:
      apiServerNetworkElb:
        arn: arn:aws:elasticloadbalancing:us-east-2:643893117298:
             loadbalancer/net/XXXXXXX/b6c7069ef42144eb
        attributes:
        - key: access_logs.s3.enabled
          value: "false"
          ...
        dnsName: XXXXXXX.elb.us-east-2.amazonaws.com
        name: 8dcf6cc2-f350-11ea-875a-0242ac11
        scheme: internet-facing
      securityGroups:
        bastion:
          id: sg-07ce2664e174f2692
          ingressRule:
          - cidrBlocks:
            - 0.0.0.0/0
            description: SSH
            fromPort: 22
            protocol: tcp
            sourceSecurityGroupIds: null
            toPort: 22
          name: 8dcf6cc2-f350-11ea-875a-0242ac110002-bastion
          tags:
            Name: 8dcf6cc2-f350-11ea-875a-0242ac110002-bastion
            kubernetes.io/cluster/8dcf6cc2-f350-11ea-875a-0242ac11000: owned
            sigs.k8s.io/cluster-api-provider-aws/managed: "true"
            sigs.k8s.io/cluster-api-provider-aws/role: bastion
        controlplane:
          id: sg-02519a43ab2067af4
          ingressRule:
            ...
        node:
          id: sg-00ae75ef271d78580
          ingressRule:
            ...

status:providerStatus for Bastion¶

Warning

Do not modify this section using API.

The bastion section of the providerStatus object field of the Cluster resource contains the Bastion instance configuration:

ebsOptimized
Indicates whether the instance is optimized for Amazon EBS I/O
enaSupport
Specifies whether enhanced networking with ENA is enabled
id
Unique identifier of an instance
imageId
ID of the AMI used to launch the instance
instanceState
Current state of the instance
privateIp
Private IPv4 address assigned to the instance
publicIp
Public IPv4 address assigned to the instance, if applicable
securityGroupIds
List of security groups IDs of the instance
subnetId
ID of the instance subnet
tags
List of key-value pairs of tags associated with the instance
type
Instance type

Configuration example:

status:
  providerStatus:
    bastion:
      ebsOptimized: false
      enaSupport: true
      id: i-08da6993be2f757a6
      imageId: ami-2581aa40
      instanceState: running
      privateIp: 10.0.1.138
      publicIp: 3.15.211.232
      securityGroupIds:
      - sg-07ce2664e174f2692
      subnetId: subnet-0627c9cbcae053fec
      tags:
        Name: 8dcf6cc2-f350-11ea-875a-0242ac110002-bastion
        kubernetes.io/cluster/8dcf6cc2-f350-11ea-875a-0242ac110002: owned
        sigs.k8s.io/cluster-api-provider-aws/managed: "true"
        sigs.k8s.io/cluster-api-provider-aws/role: bastion
      type: t2.micro

status:providerStatus for Cluster readiness¶

Warning

Do not modify this section using API.

The providerStatus object field of the Cluster resource that reflects cluster readiness contains the following fields:

persistentVolumesProviderProvisioned
Provision status of the provider persistent volumes (PVs). Used to prevent Helm releases that require PVs from being installed until some default StorageClass is present in the cluster.
helm
Status of deployed Helm releases:
- ready
  If all Helm releases have been deployed successfully, the value switches to true.
- releases
  List of enabled Helm Releases that run on a cluster:
  
  releaseStatuses
  List of Helm releases being deployed. Each release has the success field that switches to true once a release is deployed.
  
  stacklight
  Status of the StackLight deployment. Contains URLs of all StackLight components.
  
  iam
  Status of the IAM deployment. Contains URLs of the keycloak and api components.
  
  decc
  Status of the remaining container cloud components. Contains URLs of the ui, cache, and proxy components.
nodes
- ready
  Number of nodes that completed deployment or update.
- requested
  Total number of nodes. If the number of ready nodes does not match the number of requested nodes, it means that a cluster is being updated.
ceph
- ready
  Ceph readiness flag.
- message
  Ceph status details.
ready
Cluster readiness flag. If true, the cluster is deployed successfully and all components are up and running.
conditions
List of objects status condition:
- type
  Object type
- ready
  Readiness flag
- message
  Status details
notReadyObjects
List of Kubernetes objects (Service, Deployment, and StatefulSet) that are not in the Ready state yet:
- Service is not ready if its external address has not been provisioned yet.
- Deployment or StatefulSet is not ready if the number of ready replicas is not equal to the number of required replicas.
Contains the name and namespace of the object and the number of ready and required replicas for controllers. If all objects are ready, the notReadyObjects list is empty.

Configuration example:

status:
  providerStatus:
    persistentVolumesProviderProvisioned: true
    helm:
      ready: true
      releases:
        decc:
          cache:
            url: >-
              https://a618e3d36d7f44f2e8d56bbcc53ffbf7-1765661812.us-east-2.elb.amazonaws.com
          proxy:
            url: >-
              http://a0d8d8966e0d24f50aead0942da92456-2114585625.us-east-2.elb.amazonaws.com:3128
          ui:
            url: >-
              https://a43fe72c644de41ae9db3cc77dd992d5-566275388.us-east-2.elb.amazonaws.com
        iam:
          api:
            url: >-
              https://a08d8bdd8553b49a88ab8e663d384001-1745154108.us-east-2.elb.amazonaws.com
          keycloak:
            url: >-
              https://a2b58b6a3ee3c4884b034fd791ebff6d-1687192379.us-east-2.elb.amazonaws.com
        releaseStatuses:
          admission-controller:
            success: true
          iam:
            success: true
          iam-controller:
            success: true
          kaas-exporter:
            success: true
          kaas-public-api:
            success: true
          kaas-ui:
            success: true
          lcm-controller:
          ...
        stacklight:
          alerta:
            url: http://172.16.248.170
          alertmanager:
            url: http://172.16.247.217
          grafana:
            url: http://172.16.248.49
          kibana:
            url: http://172.16.245.164
          prometheus:
            url: http://172.16.249.211
          success: true
    nodes:
      ready: 3
      requested: 3
    notReadyObjects:
      services:
        - name: testservice
          namespace: default
      deployments:
        - name: <provider_name>-provider
          namespace: kaas
          replicas: 3
          readyReplicas: 2
      statefulsets: {}
    ready: false
    ceph:
      - message: Ceph cluster has been configured successfully
        ready: true
    conditions:
      - message: Helm charts are successfully installed(upgraded).
        ready: true
        type: Helm
      - message: Kubernetes objects are fully up.
        ready: true
        type: Kubernetes
      - message: All requested nodes are ready.
        ready: true
        type: Nodes

status:providerStatus for Open ID Connect¶

Warning

Do not modify this section using API.

The oidc section of the providerStatus object field of the Cluster resource reflects the Open ID Connect (OIDC) configuration details. It contains the required details to obtain a cluster token and contains the following fields:

certificate
Base64-encoded OIDC certificate.
clientId
Client ID for OIDC requests.
groupsClaim
Name of an OIDC groups claim.
issuerUrl
Isuer URL to get the representation of the realm.
ready
OIDC status relevance. Is true if the status fits the configuration of the LCMCluster OIDC.

Configuration example:

status:
  providerStatus:
    oidc:
      certificate: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUREekNDQWZ...
      clientId: kaas
      groupsClaim: iam_roles
      issuerUrl: https://172.16.243.211/auth/realms/iam
      ready: true

status:providerStatus for Cluster releases¶

Warning

Do not modify this section using API.

The releaseRefs section of the providerStatus object field of the Cluster resource provides the current Cluster release version as well as the one available for upgrade. It contains the following fields:

current
Details of the currently installed Cluster release:
- lcmType
  Type of the Cluster release (mke)
- name
  Name of the Cluster release resource
- version
  Release version
- unsupportedSinceKaaSVersion
  Indicates that a newer Container Cloud release exists and it does not support the current Cluster release
available
List of releases available for upgrade that contains the name and version fields

Configuration example:

status:
  providerStatus:
    releaseRefs:
      available:
        - name: mke-5-15-0-3-4-0-dev
          version: 5.15.0+3.4.0-dev
      current:
        lcmType: mke
        name: mke-5-14-0-3-3-0-beta1
        version: 5.14.0+3.3.0-beta1