k0s configuration management

Applies to management clusters only

This section describes the configuration specifics of a k0s-based cluster deployed using MOSK. For the configuration of MKE-based clusters, see MKE configuration management.

Configuration model

Unlike MKE, k0s has no configuration API that you can call directly. The Kubernetes underlay configuration of a k0s-based cluster is fully declarative and is managed as follows:

  1. MOSK composes the cluster-wide k0s configuration from its own defaults and the parameters set in the Cluster object of the cluster.

  2. The resulting configuration is stored in the spec.k0sConfig section of the LCMCluster object of the cluster.

  3. The LCM Agent renders this configuration into a static k0s configuration file on every control plane node.

Important

Editing the k0s configuration on a node or through the k0s cluster configuration object has no persistent effect: any local modification is reverted on the next reconciliation of the cluster.

Use the Cluster object as the only supported way of changing the Kubernetes underlay configuration.

Warning

Any change in the k0s configuration on a management cluster restarts the k0scontroller systemd service, which supervises all Kubernetes control plane components. MOSK cordons and drains one control plane node at a time, so the pods running on it are rescheduled. Therefore, plan the change as a maintenance operation and make sure that all cluster nodes are Ready before applying it.

Parameters managed by MOSK

MOSK sets the following k0s parameters and reapplies them on every reconciliation. The values in the tables are the MOSK defaults for a management cluster.

You can adjust the parameters marked with the icon through the generic extraArgs and spec.providerSpec.value.workerProfile.config parameters described in Parameters exposed in the Cluster object. When the Comments column names a dedicated Cluster object parameter, that parameter is the only supported way of changing the default value.

Warning

Mirantis does not recommend changing any of the defaults listed below. Since the values from the Cluster object take precedence, the generic extraArgs parameters can also override the unmarked extraArgs:* defaults. This can break the cluster or make it less secure. Contact Mirantis support before overriding an unmarked parameter.

spec:api

k0s parameter

Default value in MOSK

Comments

port

12388

Local port of the Kubernetes API server. External clients reach the cluster API through the load balancer on port 443.

k0sApiPort

4443

extraArgs:service-node-port-range

30000-32767

extraArgs:profiling

false

Set through the spec.providerSpec.value.profiling.enabled parameter of the Cluster object, together with the same argument of the controller manager and the scheduler. For configuration steps, see Configure Kubernetes auditing and profiling.

extraArgs:audit-log-*

Auditing enabled

Set through the spec.providerSpec.value.audit.kubernetes section of the Cluster object. For configuration steps, see Configure Kubernetes auditing and profiling.

extraArgs:oidc-*

Derived from the Keycloak configuration of the cluster

Managed by MOSK. Do not override.

extraArgs:encryption-provider-config

/var/lib/k0s/encryption-config.yaml

extraArgs:enable-admission-plugins

Derived from the Kubernetes version set in clusterrelease.spec.description

For details about admission plugins, see Kubernetes documentation: Admission Control in Kubernetes.

For the list of admission plugins supported by a specific Kubernetes version, see the corresponding Kubernetes release documentation.

extraArgs:service-account-lookup

true

Makes the Kubernetes API server verify that a ServiceAccount token still has its backing object in etcd, so that deleting the object revokes the token immediately. Disabling it makes token revocation less secure.

extraArgs:request-timeout

1m0s

sans

Derived from the cluster load balancer and node addresses

Managed by MOSK. To add a custom host name to the Kubernetes API endpoint, use a custom TLS certificate instead. See Configure TLS certificates for cluster applications.

spec:controllerManager and spec:scheduler

k0s parameter

Default value in MOSK

Comments

extraArgs:leader-elect-lease-duration

120s

Set for both components.

extraArgs:leader-elect-renew-deadline

60s

Set for both components.

extraArgs:profiling

false

Set through the spec.providerSpec.value.profiling.enabled parameter of the Cluster object for both components, together with the same argument of the Kubernetes API server. For configuration steps, see Configure Kubernetes auditing and profiling.

extraArgs:allocate-node-cidrs

false

Applies to controllerManager. Node CIDR allocation is delegated to Calico.

spec:storage

k0s parameter

Default value in MOSK

Comments

type

etcd

etcd:extraArgs:enable-v2

false

Managed by MOSK for automated migration from v2 to v3 of the etcd cluster. Do not override.

etcd:extraArgs:quota-backend-bytes

Unset (etcd default of 2 GB)

Set through the spec.providerSpec.value.etcd.storageQuota parameter of the Cluster object. For configuration steps, see Increase storage quota for etcd.

spec:network

k0s parameter

Default value in MOSK

Comments

provider

calico

podCIDR

10.233.64.0/18

Set in spec.clusterNetwork.pods.cidrBlocks of the Cluster object.

serviceCIDR

10.233.0.0/18

Set in spec.clusterNetwork.services.cidrBlocks of the Cluster object.

calico:mode

vxlan

calico:overlay

Always

calico:mtu

1450

Set through the spec.providerSpec.value.calico.mtu parameter of the Cluster object. For configuration steps, see Set the MTU size for Calico.

calico:vxlanPort

4792

calico:vxlanVNI

10000

calico:wireguard

false

Set through the spec.providerSpec.value.secureOverlay parameter of the Cluster object.

spec:workerProfiles

k0s parameter

Default value in MOSK

Comments

default and manager profiles

Configuration of kubelet aligned with the MKE defaults, including the reserved resources, protectKernelDefaults, the pod limits, and the container log settings.

MOSK manages two worker profiles. A common kubelet configuration that you define in spec.providerSpec.value.workerProfile.config of the Cluster object is merged into both of them. For details, see Parameters exposed in the Cluster object.

Parameters exposed in the Cluster object

To adjust the Kubernetes underlay configuration of a k0s-based cluster, use the spec.providerSpec.value section of its Cluster object. The following parameters are exposed:

  • api.extraArgs

  • controllerManager.extraArgs

  • scheduler.extraArgs

  • etcd.extraArgs

  • workerProfile.config

  • featureGates

For the detailed description of these parameters, see API Reference: Cluster resource.

Configuration example:

spec:
  ...
  providerSpec:
    value:
      ...
      api:
        extraArgs:
          max-requests-inflight: "600"
      controllerManager:
        extraArgs:
          concurrent-deployment-syncs: "10"
      scheduler:
        extraArgs:
          kube-api-qps: "100"
      etcd:
        extraArgs:
          snapshot-count: "20000"
      workerProfile:
        config:
          imageGCHighThresholdPercent: 80
          imageGCLowThresholdPercent: 70
      featureGates:
        - name: ContainerStopSignals
          enabled: true
          components:
            - kube-apiserver
            - kubelet

For the detailed description of these parameters, see API Reference: Cluster resource.

The values that you define are merged on top of the MOSK defaults, and the values from the Cluster object take precedence over the defaults. Feature gates are appended after the MOSK-managed ones, so a user-defined entry overrides a managed one with the same name.

Important

Mirantis cannot guarantee the expected behavior of the functionality configured through these parameters as long as the customer-specific configuration does not undergo testing within MOSK. Therefore, Mirantis recommends that you test custom settings on a staging environment before applying them to production.

The parameters that MOSK requires for the correct operation of the cluster, such as the OIDC and encryption provider settings, must not be overridden.

See also