Configure Alertmanager integrations¶

This section describes how to enable StackLight LMA to send notifications to specific receivers such as PagerDuty, OpsGenie, and so on using the Alertmanager service on an existing MCP cluster. For a list of supported receivers, see Prometheus Alertmanager documentation: Receiver.

The changes performed within the following procedure are backward compatible with previous MCP releases. Therefore, you do not need to change any of the already configured Alertmanager receivers and routes.

To enable Alertmanager integrations:

1. Open your project Git repository with the Reclass model on the cluster level.

2. Open the <classes/cluster/<cluster>/stacklight/server.yml> file for editing.

3. Configure the Alertmanager receiver route as required:

   parameters:
     prometheus:
       alertmanager:
         enabled: true
         config:
           route:
             routes:
               <route_name>:
                 receiver: <receiver_name>
                 match_re:
                   - label: '<name_of_the_alert_label>'
                     value: '<regex_to_identify_the_route>'
                 continue: true
           receiver:
             <receiver_name>:
               enabled: true
               generic_configs:  # <- here is the difference
                 <chosen_Alertmanager_receiver_type>:
                   <receiver_endpoint_name>:
                     <receiver_config>
   

   Parameters description¶

   Parameter name	Description
   routes	Specify a unique route name.
   receiver	Specify a unique receiver name.
   match_re	In label, specify the name of the label to filter the alerts. In value, specify a regular expression that Alertmanager will use to route the alert.
   receiver	Specify a unique receiver name. Set to the same value as routes.
   generic_configs	Specify one of the available Alertmanager receiver types. Specify a unique receiver endpoint name. Specify the endpoint configuration in the YAML format.

   Example configuration:

   parameters:
     prometheus:
       alertmanager:
         enabled: true
         config:
           route:
             routes:
               opsgenie:
                 receiver: HTTP-opsgenie
                 match_re:
                   - label: route
                     value: '(.*opsgenie.*)'
                 continue: true
           receiver:
             HTTP-opsgenie:
               enabled: true
               generic_configs:
                 opsgenie_configs:
                   opsgenie-endpoint:
                     api_url: "https://example.app.eu.opsgenie.com/"
                     api_key: "opsgeniesecretkey"
                     send_resolved: true
   

4. If you have previously configured email notifications through Alertmanager, verify that the prometheus_server_alert_label_route parameter also includes the receiver that you are configuring. For example:

   parameters:
     _param:
       prometheus_server_alert_label_route: email;opsgenie;
   

5. Log in to the Salt Master node.

6. Verify the Reclass model:

   reclass -i
   

   The output should not include any errors.

7. Verify that the changes render properly:

   salt '*mon*' state.sls prometheus.alertmanager test=True
   

   Example of system response for the configuration provided above:

   routes:
     - receiver: default
   [...]
     # opsgenie
     - receiver: HTTP-opsgenie
       continue: True
       match_re:
         route: '(.*opsgenie.*)'
   [...]
   receivers:
     - name: 'default'
   [...]
     - name: 'HTTP-opsgenie'
       opsgenie_configs:
       # opsgenie-endpoint
       -
         api_key: opsgeniesecretkey
         api_url: https://example.app.eu.opsgenie.com/
         send_resolved: true
   [...]
   

8. Update the Prometheus and Alertmanager configuration:

   salt '*mon*' state.sls prometheus.server -b 1
   salt '*mon*' state.sls prometheus.alertmanager