Alerting

Setup an alerting rule

Using AlertManager, you can get alerted when your applications don’t meet your desired requirements. This configuration creates a slack notification on the selected channel once the severity isn’t “none”.

Make sure the according PrometheusRule has been created in order to let this alert fire.

alert-config.yaml

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: slack-alert-config
  namespace: monitoring
spec:
  route:
    receiver: 'slack-notifications'
    groupBy: ['alertname', 'severity']
    groupWait: 30s
    groupInterval: 5m
    repeatInterval: 4h
    matchers:
    - name: severity
      value: none
      matchType: '!='
  receivers:
    - name: 'slack-notifications'
      slackConfigs:
        - sendResolved: true
          apiURL:
            key: SLACK_WEBHOOK_URL
            name: slack-webhook-secret
          channel: '#alerts'
          title: '{{ .CommonAnnotations.summary }}'
          text: '{{ .CommonAnnotations.description }}'