Skip to content

Using KOF (without Grafana)#

Optional Grafana, Mirantis Grafana Dashboards, and grafana-operator#

Note

Using Grafana with KOF

Effective immediately, Mirantis will no longer distribute Grafana as part of its products or services. This change is being made to proactively avoid potential licensing, redistribution, or compliance considerations related to third-party software. For more information, please contact Mirantis.

If you want to install and enable Grafana and use the Mirantis-provided Grafana dashboards, please see Grafana in KOF.

This document explains how to use KOF without Grafana, using other provided user interfaces.

Metrics#

  • Prometheus UI:
    • Run in the management cluster: 
      kubectl port-forward -n kof svc/kof-mothership-promxy 8082:8082
    • Explore the Graph: http://127.0.0.1:8082/graph?g0.expr=up&g0.tab=0
    • CLI queries for automation: 
      curl http://localhost:8082/api/v1/query?query=up \
  | jq '.data.result | map(.metric.cluster) | unique'

curl http://localhost:8082/api/v1/query?query=up \
  | jq '.data.result | map(.metric.job) | unique'

curl http://localhost:8082/api/v1/query \
  -d 'query=up{cluster="mothership", job="kof-collectors-opencost"}' \
  | jq
  • VictoriaMetrics UI:

Logs#

  • VictoriaLogs UI:
    • Run in the regional cluster: 
      KUBECONFIG=regional-kubeconfig kubectl port-forward \
  -n kof svc/kof-storage-victoria-logs-cluster-vlselect 9471:9471
      We're using port 9471, not 9428.
    • Open http://127.0.0.1:9471/select/vmui/
    • CLI query for automation: 
      curl http://127.0.0.1:9471/select/logsql/query \
  -d 'query=_time:1h' \
  -d 'limit=10'
  • Run inside of Istio mesh: 
    curl http://$REGIONAL_CLUSTER_NAME-logs-select:9471/select/logsql/query \
  -d 'query=_time:1h' \
  -d 'limit=10'
  • Run without Istio and port-forwarding: 
    VM_USER=$(
  kubectl get secret -n kof storage-vmuser-credentials -o yaml \
  | yq .data.username | base64 -d
)
VM_PASS=$(
  kubectl get secret -n kof storage-vmuser-credentials -o yaml \
  | yq .data.password | base64 -d
)
curl https://vmauth.$REGIONAL_DOMAIN/vls/select/logsql/query \
  -u "$VM_USER":"$VM_PASS" \
  -d 'query=_time:1h' \
  -d 'limit=10'

Access to Jaeger#

Jaeger UI of each regional cluster can be accessed by following these steps:

  1. Ensure you have the regional-kubeconfig file created on the verification step.

  2. If you've applied the Istio section:

    • Forward a port to the Jaeger UI: 

      KUBECONFIG=regional-kubeconfig kubectl port-forward \
  -n kof svc/kof-storage-jaeger-query 16686:16686

    • Open the link http://127.0.0.1:16686/search and explore the Jaeger UI.

  3. If you have not applied the Istio section:

    • Ensure you have the REGIONAL_DOMAIN variable set on the installation step.

    • Get the regional Jaeger username and password: 

      KUBECONFIG=regional-kubeconfig kubectl get secret \
  -n kof jaeger-credentials -o yaml | yq '{
  "user": .data.username | @base64d,
  "pass": .data.password | @base64d
}'

    • Get the the Jaeger UI URL, open it, and login with the username/password printed above: 

      echo https://jaeger.$REGIONAL_DOMAIN

Cost Management (OpenCost)#

KOF includes OpenCost, which provides cost management features for Kubernetes clusters. Common metrics (also available in the pre-installed Grafana FinOps dashboards if enabled) are:

  • node_total_hourly_cost (per-node hourly cost)
  • Namespace and pod-level cost allocation
  • Historical spend trends and efficiency ratios
Metric Description
node_total_hourly_cost Hourly cost per node (includes CPU, memory, storage)
namespace_cpu_cost CPU cost aggregated by namespace
namespace_memory_cost Memory cost aggregated by namespace
pod_cost Cost allocation at pod granularity
cluster_efficiency Ratio of requested vs actual resource usage

Once you have this information, you can optimize your cluster. Typical optimizations include:

  • Identify under-utilized resources and right-size workloads