Cluster¶
This section describes the Cluster
resource used the in Mirantis
Container Cloud API that describes the cluster-level parameters.
For demonstration purposes, the Container Cloud Cluster
custom resource (CR) is split into the following major sections:
Warning
The fields of the Cluster
resource that are located
under the status
section including providerStatus
are available for viewing only.
They are automatically generated by the bare metal cloud provider
and must not be modified using Container Cloud API.
metadata¶
The Container Cloud Cluster
CR contains the following fields:
apiVersion
API version of the object that is
cluster.k8s.io/v1alpha1
.
kind
Object type that is
Cluster
.
The metadata
object field of the Cluster
resource
contains the following fields:
name
Name of a cluster. A managed cluster name is specified under the
Cluster Name
field in the Create Cluster wizard of the Container Cloud web UI. A management cluster name is configurable in the bootstrap script.
namespace
Project in which the cluster object was created. The management cluster is always created in the
default
project. The managed cluster project equals to the selected project name.
labels
Key-value pairs attached to the object:
kaas.mirantis.com/provider
Provider type that is
baremetal
for the baremetal-based clusters.
kaas.mirantis.com/region
Region name. The default region name for the management cluster is
region-one
.Note
The
kaas.mirantis.com/region
label is removed from all Container Cloud objects in 2.26.0 (Cluster releases 17.1.0 and 16.1.0). Therefore, do not add the label starting these releases. On existing clusters updated to these releases, or if manually added, this label will be ignored by Container Cloud.
Warning
Labels and annotations that are not documented in this API Reference are generated automatically by Container Cloud. Do not modify them using the Container Cloud API.
Configuration example:
apiVersion: cluster.k8s.io/v1alpha1
kind: Cluster
metadata:
name: demo
namespace: test
labels:
kaas.mirantis.com/provider: baremetal
spec:providerSpec¶
The spec
object field of the Cluster
object
represents the BaremetalClusterProviderSpec
subresource that
contains a complete description of the desired bare metal cluster
state and all details to create the cluster-level
resources. It also contains the fields required for LCM deployment
and integration of the Container Cloud components.
The providerSpec
object field is custom for each cloud provider and
contains the following generic fields for the bare metal provider:
apiVersion
API version of the object that is
baremetal.k8s.io/v1alpha1
kind
Object type that is
BaremetalClusterProviderSpec
Configuration example:
spec:
...
providerSpec:
value:
apiVersion: baremetal.k8s.io/v1alpha1
kind: BaremetalClusterProviderSpec
spec:providerSpec common¶
The common providerSpec
object field of the Cluster
resource
contains the following fields:
credentials
Field reserved for other cloud providers, has an empty value. Disregard this field.
spec:providerSpec common¶
The common providerSpec
object field of the Cluster
resource
contains the following fields:
credentials
Field reserved for other cloud providers, has an empty value. Disregard this field.
release
Name of the
ClusterRelease
object to install on a cluster
helmReleases
List of enabled Helm releases from the
Release
object that run on a cluster
proxy
Name of the
Proxy
object
tls
TLS configuration for endpoints of a cluster
keycloak
KeyCloak endpoint
tlsConfigRef
Reference to the
TLSConfig
object
ui
Web UI endpoint
tlsConfigRef
Reference to the
TLSConfig
object
For more details, see TLSConfig resource.
maintenance
Maintenance mode of a cluster. Prepares a cluster for maintenance and enables the possibility to switch machines into maintenance mode.
containerRegistries
List of the
ContainerRegistries
resources names.
ntpEnabled
NTP server mode. Boolean, enabled by default.
Since Container Cloud 2.23.0, you can optionally disable NTP to disable the management of
chrony
configuration by Container Cloud and use your own system forchrony
management. Otherwise, configure the regional NTP server parameters to be applied to all machines of managed clusters.Before Container Cloud 2.23.0, you can optionally configure NTP parameters if servers from the Ubuntu NTP pool (
*.ubuntu.pool.ntp.org
) are accessible from the node where a management cluster is being provisioned. Otherwise, this configuration is mandatory.NTP configuration
Configure the regional NTP server parameters to be applied to all machines of managed clusters.
In the
Cluster
object, add thentp:servers
section with the list of required server names:spec: ... providerSpec: value: kaas: ... ntpEnabled: true regional: - helmReleases: - name: <providerName>-provider values: config: lcm: ... ntp: servers: - 0.pool.ntp.org ... provider: <providerName> ...
To disable NTP:
spec: ... providerSpec: value: ... ntpEnabled: false ...
audit
Since 2.24.0 as TechPreviewOptional. Auditing tools enabled on the cluster. Contains the
auditd
field that enables the Linux Audit daemon auditd to monitor activity of cluster processes and prevent potential malicious activity.Configuration for auditd
In the
Cluster
object, add the auditd parameters:spec: providerSpec: value: audit: auditd: enabled: <bool> enabledAtBoot: <bool> backlogLimit: <int> maxLogFile: <int> maxLogFileAction: <string> maxLogFileKeep: <int> mayHaltSystem: <bool> presetRules: <string> customRules: <string> customRulesX32: <text> customRulesX64: <text>
Configuration parameters for auditd:
enabled
Boolean, default -
false
. Enables theauditd
role to install the auditd packages and configure rules. CIS rules: 4.1.1.1, 4.1.1.2.enabledAtBoot
Boolean, default -
false
. Configures grub to audit processes that can be audited even if they start up prior to auditd startup. CIS rule: 4.1.1.3.backlogLimit
Integer, default - none. Configures the backlog to hold records. If during boot
audit=1
is configured, the backlog holds 64 records. If more than 64 records are created during boot, auditd records will be lost with a potential malicious activity being undetected. CIS rule: 4.1.1.4.maxLogFile
Integer, default - none. Configures the maximum size of the audit log file. Once the log reaches the maximum size, it is rotated and a new log file is created. CIS rule: 4.1.2.1.
maxLogFileAction
String, default - none. Defines handling of the audit log file reaching the maximum file size. Allowed values:
keep_logs
- rotate logs but never delete themrotate
- add a cron job to compress rotated log files and keep maximum 5 compressed files.compress
- compress log files and keep them under the/var/log/auditd/
directory. Requiresauditd_max_log_file_keep
to be enabled.
CIS rule: 4.1.2.2.
maxLogFileKeep
Integer, default -
5
. Defines the number of compressed log files to keep under the/var/log/auditd/
directory. Requiresauditd_max_log_file_action=compress
. CIS rules - none.mayHaltSystem
Boolean, default -
false
. Halts the system when the audit logs are full. Applies the following configuration:space_left_action = email
action_mail_acct = root
admin_space_left_action = halt
CIS rule: 4.1.2.3.
customRules
String, default - none. Base64-encoded content of the
60-custom.rules
file for any architecture. CIS rules - none.customRulesX32
String, default - none. Base64-encoded content of the
60-custom.rules
file for thei386
architecture. CIS rules - none.customRulesX64
String, default - none. Base64-encoded content of the
60-custom.rules
file for thex86_64
architecture. CIS rules - none.presetRules
String, default - none. Comma-separated list of the following built-in preset rules:
access
actions
delete
docker
identity
immutable
logins
mac-policy
modules
mounts
perm-mod
privileged
scope
session
system-locale
time-change
Since Container Cloud 2.28.0 (Cluster releases 17.3.0 and 16.3.0) in the Technology Preview scope, you can collect some of the preset rules indicated above as groups and use them in
presetRules
:ubuntu-cis-rules
- this group contains rules to comply with the Ubuntu CIS Benchmark recommendations, including the following CIS Ubuntu 20.04 v2.0.1 rules:scope
- 5.2.3.1actions
- same as 5.2.3.2time-change
- 5.2.3.4system-locale
- 5.2.3.5privileged
- 5.2.3.6access
- 5.2.3.7identity
- 5.2.3.8
perm-mod
- 5.2.3.9mounts
- 5.2.3.10session
- 5.2.3.11logins
- 5.2.3.12delete
- 5.2.3.13mac-policy
- 5.2.3.14modules
- 5.2.3.19
docker-cis-rules
- this group contains rules to comply with Docker CIS Benchmark recommendations, including thedocker
Docker CIS v1.6.0 rules 1.1.3 - 1.1.18.
You can also use two additional keywords inside
presetRules
:none
- select no built-in rules.all
- select all built-in rules. When using this keyword, you can add the!
prefix to a rule name to exclude some rules. You can use the!
prefix for rules only if you add theall
keyword as the first rule. Place a rule with the!
prefix only after theall
keyword.
Example configurations:
presetRules: none
- disable all preset rulespresetRules: docker
- enable only thedocker
rulespresetRules: access,actions,logins
- enable only theaccess
,actions
, andlogins
rulespresetRules: ubuntu-cis-rules
- enable all rules from theubuntu-cis-rules
grouppresetRules: docker-cis-rules,actions
- enable all rules from thedocker-cis-rules
group and theactions
rulepresetRules: all
- enable all preset rulespresetRules: all,!immutable,!sessions
- enable all preset rules exceptimmutable
andsessions
CIS controls
4.1.3 (time-change
)4.1.4 (identity
)4.1.5 (system-locale
)4.1.6 (mac-policy
)4.1.7 (logins
)4.1.8 (session
)4.1.9 (perm-mod
)4.1.10 (access
)4.1.11 (privileged
)4.1.12 (mounts
)4.1.13 (delete
)4.1.14 (scope
)4.1.15 (actions
)4.1.16 (modules
)4.1.17 (immutable
)Docker CIS controls
1.1.41.1.81.1.101.1.121.1.131.1.151.1.161.1.171.1.181.2.31.2.41.2.51.2.61.2.71.2.101.2.11
secureOverlay
Optional. Technology Preview. Deprecated since Container Cloud 2.29.0 (Cluster releases 17.4.0 and 16.4.0). Available since Container Cloud 2.24.0 (Cluster release 14.0.0). Enables WireGuard for traffic encryption on the Kubernetes workloads network. Boolean. Disabled by default.
Caution
Before enabling WireGuard, ensure that the Calico MTU size is at least 60 bytes smaller than the interface MTU size of the workload network. IPv4 WireGuard uses a 60-byte header. For details, see Set the MTU size for Calico.
Caution
Changing this parameter on a running cluster causes a downtime that can vary depending on the cluster size.
For more details about WireGuard, see Calico documentation: Encrypt in-cluster pod traffic.
useBGPAnnouncement
Since 2.24.4 as TechPreviewOptional. To enable the use of BGP announcement for the cluster API LB address, set to
true
. See Configure BGP announcement for cluster API LB address for details.
Configuration example:
spec:
...
providerSpec:
value:
credentials: ""
publicKeys:
- name: bootstrap-key
release: ucp-5-7-0-3-3-3-tp11
helmReleases:
- name: metallb
values:
configInline:
address-pools:
- addresses:
- 10.0.0.101-10.0.0.120
name: default
protocol: layer2
...
- name: stacklight
...
tls:
keycloak:
certificate:
name: keycloak
hostname: container-cloud-auth.example.com
ui:
certificate:
name: ui
hostname: container-cloud-ui.example.com
containerRegistries:
- demoregistry
ntpEnabled: false
...
spec:providerSpec configuration¶
This section represents the Container Cloud components that are enabled on a cluster. It contains the following fields:
management
Configuration for the management cluster components:
enabled
Management cluster enabled (
true
) or disabled (false
).
helmReleases
List of the management cluster Helm releases that will be installed on the cluster. A Helm release includes the
name
andvalues
fields. The specified values will be merged with relevant Helm release values of the management cluster in theRelease
object.
regional
List of regional cluster components for the provider:
provider
Provider type that is
baremetal
.
helmReleases
List of the regional Helm releases that will be installed on the cluster. A Helm release includes the
name
andvalues
fields. The specified values will be merged with relevant regional Helm release values in theRelease
object.
release
Name of the Container Cloud
Release
object.
Configuration example:
spec:
...
providerSpec:
value:
kaas:
management:
enabled: true
helmReleases:
- name: kaas-ui
values:
serviceConfig:
server: https://10.0.0.117
regional:
- helmReleases:
- name: baremetal-provider
values: {}
provider: baremetal
...
release: kaas-2-0-0
status:providerStatus common¶
Must not be modified using API
The common providerStatus
object field of the Cluster
resource
contains the following fields:
apiVersion
API version of the object that is
baremetal.k8s.io/v1alpha1
kind
Object type that is
BaremetalClusterProviderStatus
loadBalancerHost
Load balancer IP or host name of the Container Cloud cluster
apiServerCertificate
Server certificate of Kubernetes API
ucpDashboard
URL of the Mirantis Kubernetes Engine (MKE) Dashboard
maintenance
Maintenance mode of a cluster. Prepares a cluster for maintenance and enables the possibility to switch machines into maintenance mode.
Configuration example:
status:
providerStatus:
apiVersion: baremetal.k8s.io/v1alpha1
kind: BaremetalClusterProviderStatus
loadBalancerHost: 10.0.0.100
apiServerCertificate: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS…
ucpDashboard: https://10.0.0.100:6443
status:providerStatus for cluster readiness¶
Must not be modified using API
The providerStatus
object field of the Cluster
resource that reflects
the cluster readiness contains the following fields:
persistentVolumesProviderProvisioned
Status of the persistent volumes provisioning. Prevents the Helm releases that require persistent volumes from being installed until some default
StorageClass
is added to theCluster
object.
helm
Details about the deployed Helm releases:
ready
Status of the deployed Helm releases. The
true
value indicates that all Helm releases are deployed successfully.
releases
List of the enabled Helm releases that run on the Container Cloud cluster:
releaseStatuses
List of the deployed Helm releases. The
success: true
field indicates that the release is deployed successfully.
stacklight
Status of the StackLight deployment. Contains URLs of all StackLight components. The
success: true
field indicates that StackLight is deployed successfully.
nodes
Details about the cluster nodes:
ready
Number of nodes that completed the deployment or update.
requested
Total number of nodes. If the number of
ready
nodes does not match the number ofrequested
nodes, it means that a cluster is being currently deployed or updated.
notReadyObjects
The list of the
services
,deployments
, andstatefulsets
Kubernetes objects that are not in theReady
state yet. Aservice
is not ready if its external address has not been provisioned yet. Adeployment
orstatefulset
is not ready if the number of ready replicas is not equal to the number of desired replicas. Both objects contain the name and namespace of the object and the number of ready and desired replicas (for controllers). If all objects are ready, thenotReadyObjects
list is empty.
Configuration example:
status:
providerStatus:
persistentVolumesProviderProvisioned: true
helm:
ready: true
releases:
releaseStatuses:
iam:
success: true
...
stacklight:
alerta:
url: http://10.0.0.106
alertmanager:
url: http://10.0.0.107
grafana:
url: http://10.0.0.108
kibana:
url: http://10.0.0.109
prometheus:
url: http://10.0.0.110
success: true
nodes:
ready: 3
requested: 3
notReadyObjects:
services:
- name: testservice
namespace: default
deployments:
- name: baremetal-provider
namespace: kaas
replicas: 3
readyReplicas: 2
statefulsets: {}
status:providerStatus for Open ID Connect¶
Must not be modified using API
The oidc
section of the providerStatus
object field
in the Cluster
resource reflects the Open ID Connect configuration details.
It contains the required details to obtain a token for
a Container Cloud cluster and consists of the following fields:
certificate
Base64-encoded OIDC certificate.
clientId
Client ID for OIDC requests.
groupsClaim
Name of an OIDC groups claim.
issuerUrl
Issuer URL to obtain the representation of the realm.
ready
OIDC status relevance. If
true
, the status corresponds to theLCMCluster
OIDC configuration.
Configuration example:
status:
providerStatus:
oidc:
certificate: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUREekNDQWZ...
clientId: kaas
groupsClaim: iam_roles
issuerUrl: https://10.0.0.117/auth/realms/iam
ready: true
status:providerStatus for cluster releases¶
Must not be modified using API
The releaseRefs
section of the providerStatus
object field
in the Cluster
resource provides the current Cluster release version
as well as the one available for upgrade. It contains the following fields:
current
Details of the currently installed Cluster release:
lcmType
Type of the Cluster release (
ucp
).
name
Name of the Cluster release resource.
version
Version of the Cluster release.
unsupportedSinceKaaSVersion
Indicates that a Container Cloud release newer than the current one exists and that it does not support the current Cluster release.
available
List of the releases available for upgrade. Contains the
name
andversion
fields.
Configuration example:
status:
providerStatus:
releaseRefs:
available:
- name: ucp-5-5-0-3-4-0-dev
version: 5.5.0+3.4.0-dev
current:
lcmType: ucp
name: ucp-5-4-0-3-3-0-beta1
version: 5.4.0+3.3.0-beta1