Skip to content

proposal: first-class alerting via PrometheusRule and Grafana #79

Description

@atkrad

Scraping is first-class (metrics: true makes a ServiceMonitor / PodMonitor). Alerting is not. Today you drop a PrometheusRule in .deployah/manifests/.

We want component-owned alerts, with Deployah labels, plan diffs, and a capability check. Two backends exist: Prometheus Operator (PrometheusRule) and Grafana Operator (GrafanaAlertRuleGroup). They are not the same CR, so we will not invent one DSL that pretends to emit both.

Platform picks the backends. The component owns the PromQL.

# deployah.platform.yaml
profiles:
  observability:
    metrics:
      monitorLabels:
        release: prometheus
    alerting:
      providers: [prometheusRule]
      prometheusRule:
        ruleLabels:
          release: prometheus
# deployah.yaml
components:
  api:
    metrics: true
    profiles: [observability]
    alerts:
      prometheusRule:
        groups:
          - name: api
            rules:
              - alert: APIHighErrorRate
                expr: |
                  sum(rate(http_requests_total{component="api",code=~"5.."}[5m]))
                    /
                  sum(rate(http_requests_total{component="api"}[5m]))
                    > 0.05
                for: 5m

v1 emits PrometheusRule only. Grafana (alerts.grafana) is phase 2. Do not run Helm tpl on expr or annotations. Do not install the operators. Extras stay valid for AlertmanagerConfig, contact points, and one-offs.

Related: #18 (scrape), extras for everything else.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/helmChart render, release, post-renderarea/specdeployah.yaml schema or loadingkind/proposalDesign discussion before or during implementation

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions