Create alert with Prometheus ¶

This guide shows you how to create alerts for your application.

Prerequisites ¶

Your application serves metrics

You can define alerts by using Kubernetes resources (PrometheusRule), as well as directly in Grafana (GUI based).

You will have a separate Alertmanager for each environment available at <<tenant_url("alertmanager.<MY-ENV>")>>.

Create PrometheusRule ¶

We use native Prometheus alert rules, and let Alertmanager handle the notifications.

You can define alerts by creating a PrometheusRule resource in your teams namespace.

yaml

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: <MY-ALERT>
  namespace: <MY-TEAM>
  labels:
    team: <MY-TEAM>
spec:
  groups:
  - name: <MY-ALERT>
    rules:
    - alert: InstanceDown
      expr: kube_deployment_status_replicas_available{namespace="<namespace>", deployment="<application name>"} == 0
      for: 5m
      annotations:
        consequence: Application is unavailable
        action: "`kubectl describe pod <podname>` -> `kubectl logs <podname>`"
        summary: |-
          This is a multi-line summary with
          linebreaks and everything. Here you can give a more detailed
          summary of what this alert is about
      labels:
        namespace: <MY-TEAM> # required
        severity: critical

Activate the alert ¶

Automatically Manually

Add the file to your application repository, alongside nais.yaml to deploy with Nais github action.

bash

kubectl apply -f ./nais/alert.yaml

Verify your alert ¶

You can see the alerts in the Alertmanager at <<tenant_url("alertmanager.<MY-ENV>")>> and the defined rules in Prometheus at <<tenant_url("prometheus.<MY-ENV>")>>

Learn more ¶

💡 Advanced alerting with Prometheus 📖 Prometheus alerts reference