Configure Alertmanager integrations

Configure Alertmanager integrations

Note

This feature is available starting from the MCP 2019.2.9 maintenance update. Before using the feature, follow the steps described in Apply maintenance updates.

This section describes how to enable StackLight LMA to send notifications to specific receivers such as PagerDuty, OpsGenie, and so on using the Alertmanager service on an existing MCP cluster. For a list of supported receivers, see Prometheus Alertmanager documentation: Receiver.

The changes performed within the following procedure are backward compatible with previous MCP releases. Therefore, you do not need to change any of the already configured Alertmanager receivers and routes.

To enable Alertmanager integrations:

  1. Open your project Git repository with the Reclass model on the cluster level.

  2. Open the <classes/cluster/<cluster>/stacklight/server.yml> file for editing.

  3. Configure the Alertmanager receiver route as required:

    parameters:
      prometheus:
        alertmanager:
          enabled: true
          config:
            route:
              routes:
                <route_name>:
                  receiver: <receiver_name>
                  match_re:
                    - label: '<name_of_the_alert_label>'
                      value: '<regex_to_identify_the_route>'
                  continue: true
            receiver:
              <receiver_name>:
                enabled: true
                generic_configs:  # <- here is the difference
                  <chosen_Alertmanager_receiver_type>:
                    <receiver_endpoint_name>:
                      <receiver_config>
    
    Parameters description

    Parameter name

    Description

    routes

    Specify a unique route name.

    receiver

    Specify a unique receiver name.

    match_re

    • In label, specify the name of the label to filter the alerts.

    • In value, specify a regular expression that Alertmanager will use to route the alert.

    receiver

    Specify a unique receiver name. Set to the same value as routes.

    generic_configs

    • Specify one of the available Alertmanager receiver types.

    • Specify a unique receiver endpoint name.

    • Specify the endpoint configuration in the YAML format.

    Example configuration:

    parameters:
      prometheus:
        alertmanager:
          enabled: true
          config:
            route:
              routes:
                opsgenie:
                  receiver: HTTP-opsgenie
                  match_re:
                    - label: route
                      value: '(.*opsgenie.*)'
                  continue: true
            receiver:
              HTTP-opsgenie:
                enabled: true
                generic_configs:
                  opsgenie_configs:
                    opsgenie-endpoint:
                      api_url: "https://example.app.eu.opsgenie.com/"
                      api_key: "opsgeniesecretkey"
                      send_resolved: true
    
  4. If you have previously configured email notifications through Alertmanager, verify that the prometheus_server_alert_label_route parameter also includes the receiver that you are configuring. For example:

    parameters:
      _param:
        prometheus_server_alert_label_route: email;opsgenie;
    
  5. Log in to the Salt Master node.

  6. Verify the Reclass model:

    reclass -i
    

    The output should not include any errors.

  7. Verify that the changes render properly:

    salt '*mon*' state.sls prometheus.alertmanager test=True
    

    Example of system response for the configuration provided above:

    routes:
      - receiver: default
    [...]
      # opsgenie
      - receiver: HTTP-opsgenie
        continue: True
        match_re:
          route: '(.*opsgenie.*)'
    [...]
    receivers:
      - name: 'default'
    [...]
      - name: 'HTTP-opsgenie'
        opsgenie_configs:
        # opsgenie-endpoint
        -
          api_key: opsgeniesecretkey
          api_url: https://example.app.eu.opsgenie.com/
          send_resolved: true
    [...]
    
  8. Update the Prometheus and Alertmanager configuration:

    salt '*mon*' state.sls prometheus.server -b 1
    salt '*mon*' state.sls prometheus.alertmanager