Configure Prometheus

Configure Prometheus

You may need to configure Prometheus, for example, to modify an existing alert. Prometheus configuration is stored in the prometheus:server section of the Reclass model.

To configure Prometheus:

  1. Log in to the Salt Master node.

  2. Configure the prometheus:server section in the classes/cluster/cluster_name/stacklight/server.yml file of the Reclass model as required.

  3. Update the Salt mine:

    salt -C 'I@salt:minion' state.sls salt.minion.grains
    salt -C 'I@salt:minion' saltutil.refresh_modules
    salt -C 'I@salt:minion' mine.update
    
  4. Apply the Salt formula:

    salt -C 'I@docker:swarm and I@prometheus:server' state.sls prometheus.server -b1
    

Example configuration:

prometheus:
  server:
    enabled: true
    bind:
      port: 9090
      address: 0.0.0.0
    storage:
      local:
        engine: "persisted"
        retention: "360h"
        memory_chunks: 1048576
        max_chunks_to_persist: 524288
        num_fingerprint_mutexes: 4096
    alertmanager:
      notification_queue_capacity: 10000
    config:
      global:
        scrape_interval: "15s"
        scrape_timeout: "15s"
        evaluation_interval: "1m"
        external_labels:
          region: 'region1'
    alert:
      PrometheusTargetDownKubernetesNodes:
        if: 'up{job="kubernetes-nodes"} != 1'
        labels:
          severity: down
          service: prometheus
        annotations:
          summary: 'Prometheus target down'

The following table describes the available settings.

Settings description
Setting Description
storage The storage YAML dictionary stores the configuration options for the Prometheus storage database. These options are passed to the Prometheus server through the command-line arguments.
config The config YAML dictionary contains the options that will be placed in the Prometheus configuration file. For more information, see Prometheus configuration documentation.
alert

The alert YAML dictionary is used to generate Prometheus alerting rules. For more information, see Alerting rules.

Alternatively, you can import alerts from the */meta/prometheus.yml file of any Salt formula. However, this should be performed only by deployment engineers or developers.

Caution

The Prometheus data directory is mounted from the Docker host. If you restart a container, it can be spawned on a different host. This can cause Prometheus to start with an empty storage. In such case, the data will still be available on the previous host.

See also

Manage alerts