Cluster resource

This section outlines the Cluster resource used the in MOSK that describes the cluster-level parameters.

For demonstration purposes, the Cluster custom resource (CR) is split into the following major sections:

Warning

The fields of the Cluster resource that are located under the status section including providerStatus are available for viewing only. They are automatically generated by the bare-metal cloud provider and must not be modified using API.

metadata

The Cluster CR contains the following fields:

  • apiVersion

    API version of the object that is cluster.k8s.io/v1alpha1.

  • kind

    Object type that is Cluster.

The metadata object field of the Cluster resource contains the following fields:

  • name

    Name of a cluster. A MOSK cluster name is specified under the Cluster Name field in the Create Cluster wizard of the MOSK management console. A management cluster name is configurable in the bootstrap script.

  • namespace

    Project in which the cluster object was created. The management cluster is always created in the default project. The MOSK cluster project equals to the selected project name.

  • labels

    Key-value pairs attached to the object:

    • kaas.mirantis.com/provider

      Provider name that is baremetal.

    Warning

    Labels and annotations that are not documented in this API Reference are generated automatically. Do not modify them using the API.

Configuration example:

apiVersion: cluster.k8s.io/v1alpha1
kind: Cluster
metadata:
  name: demo
  namespace: test
  labels:
    kaas.mirantis.com/provider: baremetal

spec:providerSpec

The spec object field of the Cluster object represents the BaremetalClusterProviderSpec subresource that contains a complete description of the desired bare-metal cluster state and all details to create the cluster-level resources. It also contains the fields required for LCM deployment and integration of MOSK components.

The providerSpec object field contains the following generic fields:

  • apiVersion

    API version of the object that is baremetal.k8s.io/v1alpha1

  • kind

    Object type that is BaremetalClusterProviderSpec

Configuration example:

spec:
  ...
  providerSpec:
    value:
      apiVersion: baremetal.k8s.io/v1alpha1
      kind: BaremetalClusterProviderSpec

spec:providerSpec common

The common providerSpec object field of the Cluster resource contains the following fields:

  • credentials

    Field reserved for other cloud providers, has an empty value. Disregard this field.

  • release

    Name of the ClusterRelease object to install on a cluster.

  • helmReleases

    List of enabled Helm releases from the Release object that run on a cluster.

  • proxy

    Name of the Proxy object.

  • tls

    TLS configuration for endpoints of a cluster.

    • k0s

      Kubernetes API endpoint of a cluster that runs k0s. Applies to management clusters only.

      • tlsConfigRef

        Reference to the TLSConfig object.

      Warning

      Switching to a custom k0s certificate is irreversible. You can update the certificate at any time, but you cannot revert the Kubernetes API endpoint to the self-signed certificate generated by the MOSK management provider. An attempt to remove the tls.k0s section from the Cluster object fails with the unsetting k0s tls spec is not allowed error.

    • keycloak

      KeyCloak endpoint.

      • tlsConfigRef

        Reference to the TLSConfig object.

    • ui

      MOSK management console endpoint.

      • tlsConfigRef

        Reference to the TLSConfig object.

    For more details, see TLSConfig resource and Configure TLS certificates for cluster applications.

  • maintenance

    Maintenance mode of a cluster. Prepares a cluster for maintenance and enables the possibility to switch machines into maintenance mode.

  • containerRegistries

    List of the ContainerRegistries resources names.

  • ntpEnabled

    NTP server mode. Boolean, enabled by default.

    You can optionally disable NTP to disable the management of chrony configuration by MOSK and use your own system for chrony management. Otherwise, configure the regional NTP server parameters to be applied to all machines of MOSK clusters.

    To configure NTP during management cluster bootstrap, see Configure optional settings. After the bootstrap, you can switch to the NTP module as a flexible way to manage different NTP settings. For details, see Configure NTP server.

  • audit

    Optional. Technology preview. Auditing tools enabled on the cluster. Contains the following fields:

    • managed

      Enables the auditd host operating system configuration module to manage auditd settings for machines of both management and MOSK clusters. The only supported value is false. The true value exists for backward compatibility and will be removed in one of the future releases.

      For module details, refer to Host operating system configuration and GitHub auditd module project.

      This field was automatically added after the management cluster update to 2.31.0 on MOSK clusters that had auditd enabled. You must migrate the auditd settings to the dedicated auditd module. For the procedure, see Migrate auditd settings from the Cluster object to the auditd module.

    • auditd Unsupported

      Enables the Linux Audit daemon auditd to monitor activity of cluster processes and prevent potential malicious activity.

      Caution

      This parameter is unsupported. Use the dedicated auditd host operating system configuration module after cluster deployment instead. On existing clusters, migrate the auditd settings to the auditd module. For details, refer to the following resources:

      Configuration for auditd
      spec:
  providerSpec:
    value:
      audit:
        auditd:
          enabled: <bool>
          enabledAtBoot: <bool>
          backlogLimit: <int>
          maxLogFile: <int>
          maxLogFileAction: <string>
          maxLogFileKeep: <int>
          mayHaltSystem: <bool>
          presetRules: <string>
          customRules: <string>
          customRulesX32: <text>
          customRulesX64: <text>

      Configuration parameters for auditd:

      enabled

      Boolean, default - false. Enables the auditd role to install the auditd packages and configure rules. CIS rules: 4.1.1.1, 4.1.1.2.

      enabledAtBoot

      Boolean, default - false. Configures grub to audit processes that can be audited even if they start up prior to auditd startup. CIS rule: 4.1.1.3.

      backlogLimit

      Integer, default - none. Configures the backlog to hold records. If during boot audit=1 is configured, the backlog holds 64 records. If more than 64 records are created during boot, auditd records will be lost with a potential malicious activity being undetected. CIS rule: 4.1.1.4.

      maxLogFile

      Integer, default - none. Configures the maximum size of the audit log file. Once the log reaches the maximum size, it is rotated and a new log file is created. CIS rule: 4.1.2.1.

      maxLogFileAction

      String, default - none. Defines handling of the audit log file reaching the maximum file size. Allowed values:

      • keep_logs - rotate logs but never delete them

      • rotate - add a cron job to compress rotated log files and keep maximum 5 compressed files.

      • compress - compress log files and keep them under the /var/log/auditd/ directory. Requires auditd_max_log_file_keep to be enabled.

      CIS rule: 4.1.2.2.

      maxLogFileKeep

      Integer, default - 5. Defines the number of compressed log files to keep under the /var/log/auditd/ directory. Requires auditd_max_log_file_action=compress. CIS rules - none.

      mayHaltSystem

      Boolean, default - false. Halts the system when the audit logs are full. Applies the following configuration:

      • space_left_action = email

      • action_mail_acct = root

      • admin_space_left_action = halt

      CIS rule: 4.1.2.3.

      customRules

      String, default - none. Base64-encoded content of the 60-custom.rules file for any architecture. CIS rules - none.

      customRulesX32

      String, default - none. Base64-encoded content of the 60-custom.rules file for the i386 architecture. CIS rules - none.

      customRulesX64

      String, default - none. Base64-encoded content of the 60-custom.rules file for the x86_64 architecture. CIS rules - none.

      presetRules

      String, default - none. Comma-separated list of the following built-in preset rules:

      • access

      • actions

      • delete

      • docker

      • identity

      • immutable

      • logins

      • mac-policy

      • modules

      • mounts

      • perm-mod

      • privileged

      • scope

      • session

      • system-locale

      • time-change

      You can also collect some of the preset rules indicated above as groups and use them in presetRules:

      • ubuntu-cis-rules - this group contains rules to comply with the Ubuntu CIS Benchmark recommendations, including the following CIS Ubuntu 20.04 v2.0.1 rules:

        • scope - 5.2.3.1

        • actions - same as 5.2.3.2

        • time-change - 5.2.3.4

        • system-locale - 5.2.3.5

        • privileged - 5.2.3.6

        • access - 5.2.3.7

        • identity - 5.2.3.8

        • perm-mod - 5.2.3.9

        • mounts - 5.2.3.10

        • session - 5.2.3.11

        • logins - 5.2.3.12

        • delete - 5.2.3.13

        • mac-policy - 5.2.3.14

        • modules - 5.2.3.19

      • docker-cis-rules - this group contains rules to comply with Docker CIS Benchmark recommendations, including the docker Docker CIS v1.6.0 rules 1.1.3 - 1.1.18.

        You can also use two additional keywords inside presetRules:

        • none - select no built-in rules.

        • all - select all built-in rules. When using this keyword, you can add the ! prefix to a rule name to exclude some rules. You can use the ! prefix for rules only if you add the all keyword as the first rule. Place a rule with the ! prefix only after the all keyword.

        Example configurations:

        • presetRules: none - disable all preset rules

        • presetRules: docker - enable only the docker rules

        • presetRules: access,actions,logins - enable only the access, actions, and logins rules

        • presetRules: ubuntu-cis-rules - enable all rules from the ubuntu-cis-rules group

        • presetRules: docker-cis-rules,actions - enable all rules from the docker-cis-rules group and the actions rule

        • presetRules: all - enable all preset rules

        • presetRules: all,!immutable,!sessions - enable all preset rules except immutable and sessions


        CIS controls
        4.1.3 (time-change)
        4.1.4 (identity)
        4.1.5 (system-locale)
        4.1.6 (mac-policy)
        4.1.7 (logins)
        4.1.8 (session)
        4.1.9 (perm-mod)
        4.1.10 (access)
        4.1.11 (privileged)
        4.1.12 (mounts)
        4.1.13 (delete)
        4.1.14 (scope)
        4.1.15 (actions)
        4.1.16 (modules)
        4.1.17 (immutable)
        Docker CIS controls
        1.1.4
        1.1.8
        1.1.10
        1.1.12
        1.1.13
        1.1.15
        1.1.16
        1.1.17
        1.1.18
        1.2.3
        1.2.4
        1.2.5
        1.2.6
        1.2.7
        1.2.10
        1.2.11
  • secureOverlay

    Optional. Technology Preview. Enables WireGuard for traffic encryption on the Kubernetes workloads network. Boolean. Disabled by default.

    On an MKE-based MOSK cluster, this parameter additionally enables the MKE-specific SecureOverlay feature that is based on IPsec. On a k0s-based management cluster, it enables WireGuard only, since MKE SecureOverlay has no k0s counterpart. For details, see k0s limitations.

    Caution

    Before enabling WireGuard, ensure that the Calico MTU size is at least 60 bytes smaller than the interface MTU size of the workload network. IPv4 WireGuard uses a 60-byte header. For details, see Set the MTU size for Calico.

    Caution

    Changing this parameter on a running cluster causes a downtime that can vary depending on the cluster size.

    For more details about WireGuard, see Calico documentation: Encrypt in-cluster pod traffic.

  • useBGPAnnouncement

    Optional. Technology preview. To enable the use of BGP announcement for the cluster API LB address, set to true. See Configure BGP announcement for cluster API LB address for details.

  • api

    Optional. Applies to management clusters only. User-defined configuration of the Kubernetes API server. Contains the following field:

    • extraArgs

      Map of the Kubernetes API server command-line argument names to their values, without the leading --. For example, event-ttl: 1440h.

    The values that you set are merged on top of the MOSK defaults. If you set a key that also has a MOSK default, your value takes precedence. All other defaults remain in effect.

  • controllerManager

    Optional. Applies to management clusters only. User-defined configuration of the Kubernetes controller manager. Contains the following field:

    • extraArgs

      Map of the Kubernetes controller manager command-line argument names to their values.

    The values that you set are merged on top of the MOSK defaults. If you set a key that also has a MOSK default, your value takes precedence. All other defaults remain in effect.

  • scheduler

    Optional. Applies to management clusters only. User-defined configuration of the Kubernetes scheduler. Contains the following field:

    • extraArgs

      Map of the Kubernetes scheduler command-line argument names to their values.

    The values that you set are merged on top of the MOSK defaults. If you set a key that also has a MOSK default, your value takes precedence. All other defaults remain in effect.

  • etcd

    Optional. Applies to management clusters only. User-defined configuration of etcd. Contains the following field:

    • extraArgs

      Map of the etcd command-line argument names to their values. To change the etcd storage quota, use the dedicated etcd:storageQuota parameter instead. See Increase storage quota for etcd.

    The values that you set are merged on top of the MOSK defaults. If you set a key that also has a MOSK default, your value takes precedence. All other defaults remain in effect.

  • workerProfile

    Optional. Applies to management clusters only. Configuration for the k0s worker profiles. Contains the following field:

    • config

      Common kubelet configuration in the format of the Kubernetes KubeletConfiguration resource that is merged into all k0s worker profiles managed by MOSK.

    The values that you set are merged on top of the MOSK defaults. If you set a key that also has a MOSK default, your value takes precedence. All other defaults remain in effect.

  • featureGates

    Optional. Applies to management clusters only. List of the Kubernetes feature gates applied to the k0s components. Each entry contains the following fields:

    • name

      Name of the feature gate.

    • enabled

      Boolean value that enables the feature gate.

    • components

      Optional. List of the Kubernetes components that the feature gate applies to. If not specified, the feature gate applies to all Kubernetes components.

    User-defined gates are appended after the MOSK-managed ones and take precedence on a name conflict.

Configuration example of spec:providerSpec common
spec:
  ...
  providerSpec:
    value:
      credentials: ""
      publicKeys:
        - name: bootstrap-key
      release: ucp-5-7-0-3-3-3-tp11
      helmReleases:
        - name: metallb
          values: {}
        ...
        - name: stacklight
          ...
      tls:
        keycloak:
          certificate:
            name: keycloak
          hostname: mosk-auth.example.com
        ui:
          certificate:
            name: ui
          hostname: mosk-ui.example.com
      containerRegistries:
      - demoregistry
      ntpEnabled: false
      ...
Configuration example of the Kubernetes underlay components on a management cluster
spec:
  providerSpec:
    value:
      api:
        extraArgs:
          event-ttl: 1440h
      controllerManager:
        extraArgs:
          concurrent-deployment-syncs: "10"
      scheduler:
        extraArgs:
          kube-api-qps: "100"
      etcd:
        extraArgs:
          snapshot-count: "20000"
      workerProfile:
        config:
          imageGCHighThresholdPercent: 80
          imageGCLowThresholdPercent: 70
      featureGates:
        - name: <featureGateName>
          enabled: true
          components:
            - kube-apiserver
            - kubelet

Warning

Changing any of the Kubernetes underlay parameters of the management cluster restarts the k0scontroller systemd service that supervises all Kubernetes control plane components on every control plane node sequentially, with a cordon and drain of one node at a time.

Important

Do not override the parameters that MOSK requires for the correct operation of the cluster, such as the OIDC and encryption provider settings. Mirantis recommends testing custom settings on a staging environment before applying them to production. For the list of the managed defaults, see k0s configuration management.

spec:providerSpec configuration

This section represents MOSK components that are enabled on a cluster. It contains the following fields:

  • management

    Configuration for the management cluster components:

    • enabled

      Management cluster components enabled (true) or disabled (false). This flag can be used to determine if the current cluster is a management cluster or a MOSK cluster.

    • helmReleases

      List of the management cluster Helm releases that will be installed on the cluster. A Helm release includes the name and values fields. The specified values will be merged with relevant Helm release values of the management cluster in the Release object.

  • regional

    List of regional cluster components for the provider:

    • provider

      Provider name that is baremetal.

    • helmReleases

      List of the regional Helm releases that will be installed on the cluster. A Helm release includes the name and values fields. The specified values will be merged with relevant regional Helm release values in the Release object.

  • release

    Name of the KaaSRelease object.

Configuration example:

spec:
  ...
  providerSpec:
     value:
       kaas:
         management:
           enabled: true
           helmReleases:
             - name: kaas-ui
               values:
                 serviceConfig:
                   server: https://10.0.0.117
         regional:
           - helmReleases:
             - name: baremetal-provider
               values: {}
             provider: baremetal
           ...
         release: kaas-2-0-0

status:providerStatus common

Must not be modified using API

The common providerStatus object field of the Cluster resource contains the following fields:

  • apiVersion

    API version of the object that is baremetal.k8s.io/v1alpha1.

  • kind

    Object type that is BaremetalClusterProviderStatus.

  • loadBalancerHost

    Load balancer IP or host name of the cluster.

  • apiServerCertificate

    Server certificate of Kubernetes API.

  • ucpDashboard

    URL of the Mirantis Kubernetes Engine (MKE) Dashboard.

  • maintenance

    Maintenance mode of a cluster. Prepares a cluster for maintenance and enables the possibility to switch machines into maintenance mode.

Configuration example:

status:
  providerStatus:
    apiVersion: baremetal.k8s.io/v1alpha1
    kind: BaremetalClusterProviderStatus
    loadBalancerHost: 10.0.0.100
    apiServerCertificate: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS…
    ucpDashboard: https://10.0.0.100:6443

status:providerStatus for cluster readiness

Must not be modified using API

The providerStatus object field of the Cluster resource that reflects the cluster readiness contains the following fields:

  • persistentVolumesProviderProvisioned

    Status of the persistent volumes provisioning. Prevents the Helm releases that require persistent volumes from being installed until some default StorageClass is added to the Cluster object.

  • helm

    Details about the deployed Helm releases:

    • ready

      Status of the deployed Helm releases. The true value indicates that all Helm releases are deployed successfully.

    • releases

      List of the enabled Helm releases that run on the cluster:

      • releaseStatuses

        List of the deployed Helm releases. The success: true field indicates that the release is deployed successfully.

      • stacklight

        Status of the StackLight deployment. Contains URLs of all StackLight components. The success: true field indicates that StackLight is deployed successfully.

  • nodes

    Details about the cluster nodes:

    • ready

      Number of nodes that completed the deployment or update.

    • requested

      Total number of nodes. If the number of ready nodes does not match the number of requested nodes, it means that a cluster is being currently deployed or updated.

  • notReadyObjects

    The list of the services, deployments, and statefulsets Kubernetes objects that are not in the Ready state yet. A service is not ready if its external address has not been provisioned yet. A deployment or statefulset is not ready if the number of ready replicas is not equal to the number of desired replicas. Both objects contain the name and namespace of the object and the number of ready and desired replicas (for controllers). If all objects are ready, the notReadyObjects list is empty.

Configuration example:

status:
  providerStatus:
    persistentVolumesProviderProvisioned: true
    helm:
      ready: true
      releases:
        releaseStatuses:
          iam:
            success: true
          ...
        stacklight:
          alerta:
            url: http://10.0.0.106
          alertmanager:
            url: http://10.0.0.107
          grafana:
            url: http://10.0.0.108
          kibana:
            url: http://10.0.0.109
          prometheus:
            url: http://10.0.0.110
          success: true
    nodes:
      ready: 3
      requested: 3
    notReadyObjects:
      services:
        - name: testservice
          namespace: default
      deployments:
        - name: baremetal-provider
          namespace: kaas
          replicas: 3
          readyReplicas: 2
      statefulsets: {}

status:providerStatus for Open ID Connect

Must not be modified using API

The oidc section of the providerStatus object field in the Cluster resource reflects the Open ID Connect configuration details. It contains the required details to obtain a token for a MOSK cluster and consists of the following fields:

  • certificate

    Base64-encoded OIDC certificate.

  • clientId

    Client ID for OIDC requests.

  • groupsClaim

    Name of an OIDC groups claim.

  • issuerUrl

    Issuer URL to obtain the representation of the realm.

  • ready

    OIDC status relevance. If true, the status corresponds to the LCMCluster OIDC configuration.

Configuration example:

status:
  providerStatus:
    oidc:
      certificate: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUREekNDQWZ...
      clientId: kaas
      groupsClaim: iam_roles
      issuerUrl: https://10.0.0.117/auth/realms/iam
      ready: true

status:providerStatus for cluster releases

Must not be modified using API

The releaseRefs section of the providerStatus object field in the Cluster resource provides the current and previous Cluster release versions as well as the one available for update. It contains the following fields:

  • actualRelease

    Version of the Cluster release installed on the cluster.

  • current

    Details of the currently installed Cluster release or, if the cluster is being updated to a newer release, the target Cluster release details:

    • lcmType

      Type of the Cluster release: ucp for MKE-based MOSK clusters and k0s for k0s-based management clusters.

    • name

      Name of the Cluster release.

    • version

      Version of the Cluster release.

    • unsupportedSinceKaaSVersion

      Indicates that a KaaSRelease newer than the current one exists and that it does not support the current ClusterRelease.

    • allowedNodeLabels

      Labels that can be assigned to cluster nodes using the spec.providerSpec.value.nodeLabels field of the Machine object.

  • previous

    Details of the currently installed Cluster release or, if the cluster is being updated to a newer release, the Cluster release installed on the cluster before the update started. Contains the same fields as the current section.

  • available

    List of Cluster releases available for update:

    • name

      Name of the available Cluster release.

    • version

      Version of the available Cluster release.

    • rebootRequired

      Boolean flag indicating whether updating to the available Cluster release requires a node reboot.

Configuration example:

status:
  providerStatus:
    releaseRefs:
      actualRelease: 20.0.0+3.7.24
      available:
      - name: mke-20-1-0-3-7-28
        rebootRequired: true
        version: 20.1.0+3.7.28
      current:
        allowedNodeLabels:
        - displayName: Stacklight
          key: stacklight
          value: enabled
        - displayName: Node Rack ID
          key: rack-id
        lcmType: ucp
        name: mke-20-0-0-3-7-24
        version: 20.0.0+3.7.24
      previous:
        allowedNodeLabels:
        - displayName: Stacklight
          key: stacklight
          value: enabled
        - displayName: Node Rack ID
          key: rack-id
        lcmType: ucp
        name: mke-20-0-0-3-7-24
        version: 20.0.0+3.7.24