Configure Prometheus long-term storage

Configure Prometheus long-term storageΒΆ

You may need to configure Prometheus long-term storage to change the external labels, scrape intervals and timeouts, and so on. Since Prometheus long-term storage and Prometheus Relay are connected, you can use the same configuration file to modify Prometheus Relay, for example, to change the bind port. The configuration of Prometheus long-term storage and Prometheus Relay is stored in the prometheus:server and prometheus:relay sections of the Reclass model.

To configure Prometheus long-term storage and Prometheus Relay:

  1. Log in to the Salt Master node.

  2. Configure the prometheus:server and prometheus:relay sections in the classes/cluster/<cluster_name>/stacklight/telemetry.yml file of the Reclass model as required.

  3. Apply the Salt formula:

    salt -C 'I@prometheus:relay' state.sls prometheus
    

Example configuration of Prometheus long-term storage:

prometheus:
 server:
   dir:
     config: /etc/prometheus
     data: /var/lib/prometheus/data
   bind:
     port: 9090
     address: 0.0.0.0
   storage:
     local:
       retention: 4320h
   config:
     global:
       scrape_interval: 30s
       scrape_timeout: 30s
       evaluation_interval: 15s
       external_labels:
         region: region1

Example configuration of Prometheus Relay:

prometheus
 relay:
   enabled: true
   bind:
     port: 8080
   client:
     timeout: 12

Note

Configuring the timeout for Prometheus Relay is supported starting from the MCP 2019.2.4 maintenance update. To obtain the feature, follow the steps described in Apply maintenance updates.