Import from Prometheus

ezgh obs alert-rules import turns each alerting rule of a Prometheus rule file into an alert rule. PromQL rules follow Prometheus’s alerting semantics, so a rule behaves as it did in Prometheus, as long as the series it reads are in the project.

ezgh obs alert-rules import prometheus/alerts.yaml --contact-point "Platform on-call" --dry-run --project web
ezgh obs alert-rules import prometheus/alerts.yaml --contact-point "Platform on-call" --project web

The file is a Prometheus rule file: groups, each with rules. For a Prometheus Operator PrometheusRule, give its spec (the part with groups). - reads standard input. Files are up to 4 MiB.

What’s converted

The rest are the defaults: enabled, noData noData (which PromQL rules don’t use), onError error, repeatInterval 4h and notifyOnResolve true. --contact-point (ID, slug or name, repeatable) is set on every rule it creates; without it, the rules notify no one.

• A rule name the file repeats, often one rule per severity, becomes <alert> (<severity>) when the severity labels tell them apart, else <alert> #2, <alert> #3.
• A group interval other than 1m, a query_offset and a limit are reported and ignored: rules are evaluated every minute, a fixed time behind ingest, with at most 1,000 instances.

What’s skipped

These are listed with the reason, and nothing is sent for them:

• Recording rules (record:). Alert on the expression they record instead.
• The @ modifier.
• A range, subquery range or offset over 1 hour, such as rate(x[2h]).
• for or keep_firing_for over 24 hours.
• Labels that are templates (severity: "{{ $labels.level }}"): an alert rule’s labels are plain text.
• Annotations that call query or use $externalURL: alert templates can’t run queries.

Every rule is then checked by Observability as any rule is, and one it refuses is reported as failed with its problems while the others go on. That includes ranges, subquery ranges and offsets that add up to more than 1 hour together, annotations with range or template, labels named alertname or starting with ezgh_ or __, and a condition too expensive to evaluate every minute.

Rules that already exist

A rule whose name is already in the project is left alone (exists). With --update, it’s replaced: its condition, for, keepFiringFor, labels and annotations, and its contact points when --contact-point is given. Its other settings (enabled, no data, errors, repeat, resolve notifications) stay as they are, so a rule changed in the console keeps them. Two rules with the same name in the project are skipped.

The result

$ ezgh obs alert-rules import rules.yaml --contact-point "Platform on-call" --project web
Warning: HighErrorRate is in the file 2 times: they're named "HighErrorRate (critical)", "HighErrorRate (warning)".
Warning: EZGH evaluates every minute; api asked for 30s.
RESULT   GROUP  NAME                      DETAIL
skipped  api    job:http_requests:rate5m  recording rules aren't supported
created  api    HighErrorRate (critical)  alr_k3f9a0x2m7qp
created  api    HighErrorRate (warning)   alr_p8d2c6v1n4wz
skipped  api    SlowRequests              reads more than 1 hour ([2h])

Results are in the file’s order: created, updated, exists, skipped or failed, and with --dry-run create, update, exists or skipped. --dry-run sends nothing. -o json prints results and a summary. The command exits 1 when any rule failed.

Metric names

Imported expressions are kept as they are, so they must name series as Observability’s Prometheus API does:

• Series sent with remote write keep their names and labels, so rules written for that Prometheus usually work unchanged. Each also has an environment label: add environment="production" to a selector to read one environment, since a rule reads all of the project’s.
• OTLP metrics are translated as Prometheus’s OTLP receiver translates them: http.server.request.duration in seconds is http_server_request_duration_seconds_bucket, …_sum and …_count. Rules written against a Prometheus that scraped the same services can name series differently.
• Scrape-only series such as up and scrape_duration_seconds exist only if your Prometheus remote-writes them.

Check a rule after importing it: ezgh obs alert-rules get <name> shows its state, and the console’s Overview charts its condition over the last 6 hours. Before importing, ezgh obs alert-rules test -f evaluates a rule file once without saving it. See ezgh obs alert-rules import for every flag.