Kubernetes distributions

Mirantis OpenStack for Kubernetes (MOSK) runs its Kubernetes underlay on two distributions, depending on the cluster type:

Mirantis Kubernetes Engine (MKE): The Kubernetes distribution that MOSK has been using historically. MKE runs the Kubernetes control plane components as containers on top of Mirantis Container Runtime (MCR) and Docker Swarm, and provides its own API, web UI, and configuration file in the TOML format.
k0s: A lightweight, fully upstream, and CNCF-certified Kubernetes distribution. k0s ships as a single self-contained binary that runs the Kubernetes control plane components as regular host processes managed by systemd. It has no Docker Swarm layer, no distribution-specific API, and no web UI.

Distribution per cluster type

The distribution of a cluster is defined by the cluster type and cannot be selected during deployment.

Cluster type	Distribution	Details
Management cluster	k0s	A new management cluster is deployed on k0s. An existing management cluster is automatically migrated from MKE to k0s during the update to MOSK management 2.32.0. For details, see Migration from MKE to k0s.
MOSK cluster	MKE	MOSK clusters keep running MKE. Their automatic transition to k0s is planned for a future release and is not affected by the management cluster migration.
Bootstrap cluster	n/a	The bootstrap cluster runs neither k0s nor MKE. It is a short-lived `kind` cluster that runs on the seed node to deploy a management cluster and is removed afterwards.

Both distributions are supported by the MOSK lifecycle management engine simultaneously, so a k0s-based management cluster manages MKE-based MOSK clusters without any additional configuration.

Identifying the distribution of a cluster

To identify the Kubernetes distribution of a cluster, inspect the lcmType field of the ClusterRelease object that the cluster is running. The value is either mke or k0s:

kubectl --kubeconfig <pathToManagementClusterKubeconfig> \
  get clusterrelease $(kubectl --kubeconfig <pathToManagementClusterKubeconfig> \
  -n <projectName> get cluster <clusterName> \
  -o jsonpath='{.spec.providerSpec.value.release}') \
  -o jsonpath='{.spec.lcmType}{"\n"}'

Alternatively, verify the distribution version reported in the status of the LCMCluster object of the cluster:

A k0s-based cluster reports status:components:k0sVersion and does not report status:components:ucpVersion
An MKE-based cluster reports status:components:ucpVersion

On a cluster node, a k0s-based control plane exposes the k0scontroller systemd service and the k0s binary in /usr/local/bin/k0s:

systemctl status k0scontroller

k0s status

k0s specifics

Compared to MKE, a k0s-based cluster differs in the following aspects that are visible to a cloud operator:

Control plane processes instead of containers: The Kubernetes API server, controller manager, scheduler, and etcd run as host processes supervised by systemd, not as containers. Use systemctl status k0scontroller and journalctl -u k0scontroller instead of the docker ps and docker logs commands to inspect a control plane.
No Docker Swarm layer: k0s does not use Docker Swarm. After the migration, cluster nodes are not Swarm members and Swarm-specific ports are not used by the Kubernetes underlay.
No distribution-specific API and UI: k0s has no counterpart of the MKE API, the MKE web UI, and the MKE client bundle. Use the MOSK management console, the Kubernetes API, and StackLight dashboards instead.
Static configuration through the Cluster object: The k0s configuration is rendered by MOSK into a static configuration file on every control plane node. There is no equivalent of the MKE TOML configuration file that you can edit directly. For the configuration options that MOSK exposes, see k0s configuration management.
Container runtime: Both distributions use the containerd instance provided by MCR. k0s does not run a bundled containerd of its own: it is configured to use the containerd of the host through the /run/containerd/containerd.sock CRI socket. What differs is what runs on top of it. MKE runs the Kubernetes control plane and its own services as Docker containers, whereas on k0s the control plane components are host processes and the workloads are managed directly through the CRI.
Dedicated MetalLB address pool for Keycloak: The Kubernetes API server obtains its OIDC issuer address from a dedicated MetalLB address pool named iam-keycloak. A management cluster requires this pool to contain exactly one IP address in the /32 CIDR notation. For details, see Configure a dedicated Keycloak address pool for a management cluster.

Note

Existing management clusters that were originally deployed on MKE keep their current Keycloak address and require no changes.

MKE feature parity

Mirantis does not claim full feature parity between MKE and k0s. The following table summarizes the capability differences that are relevant for MOSK.

Capability	MKE	k0s	Details
Kubernetes distribution web UI	Supported	Unsupported	Use the MOSK management console, kubectl, and StackLight dashboards. For reference, see Access the management console, View Grafana dashboards, and View OpenSearch Dashboards.
Distribution-specific configuration API	Supported	Unsupported	The k0s configuration is fully managed by MOSK through the `Cluster` object. For reference, see k0s configuration management.
Docker Swarm workloads	Supported	Unsupported	Not applicable to MOSK, which does not run Docker Swarm workloads but Kubernetes workloads only.
Built-in LDAP integration	Supported	Unsupported	Use the external LDAP integration through Keycloak. For details, see Integration with Enterprise Identity Directory (LDAP/AD).
Mirantis Secure Registry integration	Supported	Unsupported	Use an external OCI-compatible registry. Not applicable to MOSK.
Workloads network encryption	Secure Overlay (IPsec) and Calico WireGuard	Calico WireGuard	The MKE-specific IPsec-based Secure Overlay is not available in k0s. The `secureOverlay` parameter of the `Cluster` object still enables Calico WireGuard encryption. For details, see k0s limitations.
Kubernetes certificates rotation	Supported	Partially supported	k0s rotates the Kubernetes component certificates on a service restart. Rotation of the cluster certificate authority is not supported. For details, see k0s limitations.

k0s limitations

Consider the following limitations of the k0s-based Kubernetes underlay.

Restart of the Kubernetes API server requires a node restart cycle

On MKE, changing the Kubernetes API server settings restarts the ucp-kube-apiserver container alone. On k0s, all control plane components are supervised by a single k0scontroller systemd service, so any change of the Kubernetes API server, controller manager, scheduler, etcd, or kubelet configuration requires a restart of that service on every control plane node.

MOSK performs such restarts sequentially through the maintenance API, cordoning and draining one control plane node at a time. Plan any configuration change of the Kubernetes underlay as a maintenance operation.

Rotation of the cluster certificate authority requires Mirantis assistance

k0s does not support online rotation of the Kubernetes cluster certificate authority (CA). Rotating the CA requires a full cluster downtime and must be performed with the assistance of Mirantis support.

The default validity period of the cluster CA generated for a new cluster is 10 years. The certificates of the Kubernetes components are rotated automatically when the k0scontroller service restarts.

Note

This limitation does not apply to custom TLS certificates for the Kubernetes API endpoint, which are supported and can be replaced without a cluster downtime. For details, see Configure TLS certificates for cluster applications.

MKE-specific IPsec Secure Overlay is not available

The MKE-specific Secure Overlay feature, which encrypts the workloads network using IPsec, has no counterpart in k0s. On a k0s-based cluster, the secureOverlay parameter of the Cluster object enables Calico WireGuard encryption only.

If your management cluster relied on the MKE Secure Overlay configured directly through the MKE API, plan the transition to Calico WireGuard as described in Update notes.

Not every MKE configuration option has a k0s counterpart

The MKE TOML configuration does not map one-to-one to the k0s configuration. The options that MOSK manages by default are re-applied automatically during the migration. However, the settings that were applied directly through the MKE API are not migrated, and some of them have no k0s equivalent at all.

Audit the MKE configuration of your management cluster before the update, as described in Update notes, and re-apply the required settings using the Cluster object after the update. For the list of the exposed options, see k0s configuration management.

The Kubernetes audit level is not exposed as a parameter

In Kubernetes, the audit level is not an argument of the Kubernetes API server but a field of the audit policy file. Therefore, the level parameter of the Cluster object applies to MKE-based clusters only and has no effect on a k0s-based cluster, where MOSK places an audit policy with the metadata level on the control plane nodes.

To use a different audit level, create your own audit policy file on every control plane node and point the Kubernetes API server to it using the api:extraArgs parameter of the Cluster object, as described in Configure Kubernetes auditing and profiling. MOSK does not manage such a file, so create it manually on every control plane node that you add or replace afterwards.

Support cycle of a k0s release is shorter than that of an MKE release

A k0s major release is supported for approximately 14 months, following the upstream Kubernetes release cadence, as opposed to the 2-year cycle of an MKE release. Mirantis delivers fixes through the official upstream k0s releases.