Troubleshoot Telemeter client alerts

This section describes the investigation and troubleshooting steps for the MOSK Telemeter client service.

TelemeterClientHAFailed

Root cause

The Telemeter client fails to federate data from Prometheus or to send data to the Telemeter server due to a very long incoming data sample. The limit-bytes parameter in the StackLight Helm release is too low.

Investigation

  1. Verify whether the logs of telemeter client contain alerts similar to msg="unable to forward results" err="the incoming sample data is too long":

    kubectl -n stacklight logs telemeter-client-<podID>

  2. Verify the current length limit established by Helm release:

    kubectl -n stacklight get pods telemeter-client-<podID> -o yaml | grep limit-bytes

Mitigation

  1. Add the following parameter to the StackLight Helm release values of the corresponding Cluster object:

    telemetry:
  telemeterClient:
    limitBytes: 4194304

  2. Wait for the telemeter-client-<podID> Pod to be recreated and the byte limit to be changed from --limit-bytes=1048576 to --limit-bytes=4194304.