You may need to configure Prometheus, for example, to modify an existing
alert. Prometheus configuration is stored in the prometheus:server section
of the Reclass model.
To configure Prometheus:
Log in to the Salt Master node.
Configure the prometheus:server section in the
classes/cluster/cluster_name/stacklight/server.yml file of the Reclass
model as required.
Update the Salt mine:
salt -C 'I@salt:minion' state.sls salt.minion.grains
salt -C 'I@salt:minion' saltutil.refresh_modules
salt -C 'I@salt:minion' mine.update
Apply the Salt formula:
salt -C 'I@docker:swarm and I@prometheus:server' state.sls prometheus.server -b1
Example configuration:
prometheus:
server:
enabled: true
bind:
port: 9090
address: 0.0.0.0
storage:
local:
engine: "persisted"
retention: "360h"
memory_chunks: 1048576
max_chunks_to_persist: 524288
num_fingerprint_mutexes: 4096
alertmanager:
notification_queue_capacity: 10000
config:
global:
scrape_interval: "15s"
scrape_timeout: "15s"
evaluation_interval: "1m"
external_labels:
region: 'region1'
alert:
PrometheusTargetDownKubernetesNodes:
if: 'up{job="kubernetes-nodes"} != 1'
labels:
severity: down
service: prometheus
annotations:
summary: 'Prometheus target down'
The following table describes the available settings.
| Setting | Description |
|---|---|
storage |
The storage YAML dictionary stores the configuration options for
the Prometheus storage database. These options are passed to the
Prometheus server through the command-line arguments. |
config |
The config YAML dictionary contains the options that will be placed
in the Prometheus configuration file. For more information, see
Prometheus configuration documentation. |
alert |
The Alternatively, you can import alerts from the |
Caution
The Prometheus data directory is mounted from the Docker
host. If you restart a container, it can be spawned on a
different host. This can cause Prometheus to start with an
empty storage. In such case, the data will still be available
on the previous host.
See also