Configure Kubernetes auditing and profiling

This section instructs you on how to configure Kubernetes auditing and profiling options using the Cluster object of your MOSK or management cluster. These options enable auditing and profiling of the Kubernetes control plane performance with specialized debugging endpoints.

The same Cluster object parameters apply to both Kubernetes distributions, Mirantis Kubernetes Engine (MKE) and k0s. The distribution defines only how the parameters are propagated to the Kubernetes components:

  • On an MKE-based MOSK cluster, the parameters are translated into the corresponding MKE configuration options listed below.

  • On a k0s-based management cluster, the parameters are translated into the command-line arguments of the Kubernetes API server and into the kubelet configuration of the k0s worker profiles. For details, see k0s configuration management.

Note

For management clusters, auditing is enabled by default.

Warning

Any change in the k0s configuration on a management cluster restarts the k0scontroller systemd service, which supervises all Kubernetes control plane components. MOSK cordons and drains one control plane node at a time, so the pods running on it are rescheduled. Therefore, plan the change as a maintenance operation and make sure that all cluster nodes are Ready before applying it.

Note

This note applies to MKE-based clusters only. A k0s-based cluster has no configuration API, so the Cluster object is the only supported way to change these options.

You can also enable audit_log_configuration using the MKE API with no MOSK overrides. However, if you enable the option using the Cluster object, use the same object to disable the option. Otherwise, if you disable the option using the MKE API, it will be overridden by MOSK and enabled again.

References:

Configure Kubernetes auditing and profiling options

  1. Open the Cluster object of your MOSK or management cluster for editing.

  2. In the spec:providerSpec:value: section:

    1. Add or configure the audit configuration. For example:

      spec:
  ...
  providerSpec:
    value:
      ...
      audit:
        kubernetes:
          level: request
          includeInSupportDump: true
          apiServer:
            enabled: true
            maxAge: <uint>
            maxBackup: <uint>
            maxSize: <uint>

      You can configure the following parameters. On an MKE-based cluster, they are also defined in the MKE configuration file.

      Note

      The option names enclosed in [] in the definitions below are the names of the corresponding MKE configuration options. These names apply to MKE-based clusters only, while the Cluster object parameters themselves apply to both Kubernetes distributions, unless stated otherwise in the definition.

      On a k0s-based cluster, apiServer:enabled enables auditing and the maxAge, maxBackup, and maxSize parameters are translated into the --audit-log-maxage, --audit-log-maxbackup, and --audit-log-maxsize arguments of the Kubernetes API server.

      • level

        Defines the value of [audit_log_configuration]level. Valid values are request and metadata.

        Note

        This option applies to MKE-based clusters only. On a k0s-based cluster, the audit level is defined by the Kubernetes audit policy that MOSK places on the control plane nodes, and the metadata level is used by default. Leave this parameter unset for such clusters and use a custom audit policy instead, as described in the step below.

      • includeInSupportDump

        Defines the value of [audit_log_configuration]support_dump_include_audit_logs. Boolean.

      • apiServer:enabled

        Defines the value of [cluster_config]kube_api_server_auditing. Boolean. If set to true but with no level set, the [audit_log_configuration]level MKE option is set to metadata.

        Note

        On a k0s-based management cluster, this option is enabled by default. When enabled, it starts the Kubernetes API server with the --audit-policy-file and --audit-log-path arguments. Kubernetes enables auditing only when both arguments are set.

      • maxAge

        Defines the value of kube_api_server_audit_log_maxage. Integer. If not set, defaults to 30.

      • maxBackup

        Defines the value of kube_api_server_audit_log_maxbackup. Integer. If not set, defaults to 10.

      • maxSize

        Defines the value of kube_api_server_audit_log_maxsize. Integer. If not set, defaults to 10.

    2. Optional. On a k0s-based cluster, set a custom audit level.

      Set a custom audit level on a k0s-based management cluster

      The audit level is not a Kubernetes API server argument but a field of the audit policy file. To use a level other than metadata, add your own policy file and point the API server to it:

      1. On every control plane node, create a policy file next to the MOSK one, for example /etc/kubernetes/audit-policy/audit-policy-request.yaml, owned by kube-apiserver:kube-apiserver with the 0644 permissions. Keep the first rule as is and set the required level, capitalized, in the second one:

        apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: None
    users:
      - system:apiserver
      - system:serviceaccount:kube-system:kube-dns
  - level: Request
    omitStages:
      - RequestReceived

        Warning

        Do not edit the default audit-policy.yaml because it is managed by MOSK. Otherwise, your changes will be lost.

      2. In the Cluster object, point the audit-policy-file argument of the Kubernetes API server to the new file. For example:

        spec:
  ...
  providerSpec:
    value:
      ...
      api:
        extraArgs:
          audit-policy-file: /etc/kubernetes/audit-policy/audit-policy-request.yaml

        MOSK applies the new argument and restarts k0scontroller on the control plane nodes one by one.

      Warning

      MOSK does not manage the custom policy file. Create it manually on every control plane node that you add or replace afterwards. Otherwise, the Kubernetes API server on such a node fails to start because the file that the audit-policy-file argument points to does not exist.

      Note

      The Kubernetes API server reads the audit policy only at startup. Therefore, editing a policy file that is already in use has no effect until the k0scontroller service restarts. To change the audit level afterwards, use one of the following options:

      • Add another policy file and set the audit-policy-file argument to the new path. This option is recommended because MOSK restarts the control plane nodes automatically, as during the initial configuration.

      • Edit the policy file in use and restart the k0scontroller service manually using systemctl restart k0scontroller on every control plane node, one node at a time, with the node being in maintenance mode. For reference, see Enable cluster and machine maintenance mode.

    3. Enable profiling:

      spec:
  ...
  providerSpec:
    value:
      ...
      profiling:
        enabled: true

      On an MKE-based cluster, enabling profiling automatically sets the following MKE configuration options:

      [cluster_config]kube_api_server_profiling_enabled
[cluster_config]kube_controller_manager_profiling_enabled
[cluster_config]kube_scheduler_profiling_enabled

      On a k0s-based cluster, enabling profiling sets the --profiling argument to true for the Kubernetes API server, controller manager, and scheduler.

Configuration specifics of MKE-based clusters

For the configuration procedure, see MKE documentation: Configure an existing MKE cluster.

While using this procedure, replace the command to upload the newly edited MKE configuration file with the following one:

curl --silent --insecure -X PUT -H "X-UCP-Allow-Restricted-API: i-solemnly-swear-i-am-up-to-no-good" -H "accept: application/toml" -H "Authorization: Bearer $AUTHTOKEN" --upload-file 'mke-config.toml' https://$MKE_HOST/api/ucp/config-toml

  • The value for the MKE_HOST variable has the <loadBalancerHost>:6443 format, where loadBalancerHost is the corresponding field in the cluster status.

  • The value for MKE_PASSWORD is taken from the ucp-admin-password-<clusterName> secret in the cluster namespace of the management cluster.

  • The value for MKE_USERNAME is always admin.

See also

Kubernetes documentation: Auditing