KOF Tracing#

KOF uses VictoriaTraces as the backend for distributed tracing. Traces are collected via OpenTelemetry and stored in VictoriaTraces, which provides efficient long-term storage and querying capabilities.

Configuration#

Tracing is enabled by default in KOF. The VictoriaTraces configuration can be customized in your kof-storage values.

For available configuration options, refer to the VictoriaTraces values: VictoriaTraces Cluster Helm Chart

Auto-instrumentation#

To get basic traces:

Apply OpenTelemetry Zero-code guide for your app's language.

For example, a Go executable can be instrumented like this:

kind: Deployment
spec:
  template:
    metadata:
      annotations:
        instrumentation.opentelemetry.io/inject-go: "true"
        instrumentation.opentelemetry.io/otel-go-auto-target-exe: "/path-to-executable"

Full Python app example and more options can be found in the advanced Traces guide.

Check that the pod was restarted and got a sidecar injected.
Verify the traces.

Instrumentation with SDK#

To get advanced traces:

Select and use OpenTelemetry SDK for your app's language.

For example, kof-operator was instrumentated with SDK in k0rdent/kof PR #1029.
If you don't see any traces

If you don't see any traces but you see traces export: exporter export timeout in kubectl logs -n kof deploy/kof-mothership-kof-operator this means kof-operator finished all its tasks before opentelemetry-operator made traces receiver ready.

You can generate a new test trace by asking kof-operator to reconcile something, for example:
```
kubectl label configmap -n kof kof-record-rules-default test-1=reconcile-and-trace
```
Verify the traces.

Accessing Traces#

Apply Using KOF - Traces or Grafana in KOF - Traces.