Contact points

A contact point is where a rule’s notifications go. Each is one of:

{ "name": "Platform team", "type": "email", "email": { "addresses": ["ops@example.com", "oncall@example.com"] } }

{ "name": "PagerBridge webhook", "type": "webhook", "webhook": { "url": "https://hooks.example.com/ezgh", "headers": { "Authorization": "Bearer …" } } }

{ "name": "#platform-alerts", "type": "slack", "slack": { "url": "https://hooks.slack.com/services/T0001/B0002/abcdef" } }

• name is 1 to 100 characters. A rule lists up to 10 contact points by ID.
• Write-only values are never returned: a webhook’s secret and header values, and the Slack URL. Answers show the webhook’s URL and host, hasSecret, the header names, and the Slack URL masked as hooks.slack.com/services/T…/…. A change that leaves them out keeps the stored ones, so a file in Git never has to carry them. They’re stored encrypted.
• A contact point that rules notify can’t be deleted (409 contact_point_in_use, listing them in the error’s details): take it out of them first. Notifications to a deleted contact point that haven’t been sent are dropped.
• Changes apply to the next delivery, including notifications already waiting to be retried.

Email

Alert email comes from EZGH Cloud Alerts <alerts@alerts.ezghcloud.com>. Add the address to your allow-list, or to your mail filters, so it isn’t treated as spam.

• Subject: [FIRING:2] API p95 latency (<project>), [RESOLVED] …, [NO DATA] …, [ERROR] … or [TEST] Test notification.
• Body (plain text and HTML): the state, rule and project, up to 20 instances with their summary, value and labels (“and N more”), a rule’s error message, and a link to the rule in the console.
• Each address gets its own message, with a one-click unsubscribe link and the List-Unsubscribe and List-Unsubscribe-Post headers, so mail clients offer to unsubscribe.

Confirmation

Any address can receive alerts, shared inboxes and lists included, once it opts in for the project. Every address confirms by email, organization members’ too.

1. Saving a contact point with an address the project hasn’t confirmed sends it a confirmation email, “Confirm alert emails for <organization>/<project>”, saying who added it.
2. Its Confirm link opens a page on alerts.ezghcloud.com with a button. The address is confirmed when the button is pressed, not when the link is opened, so mail scanners that open links don’t confirm it.
3. The address then receives alerts from every contact point in the project that lists it.

• The link works for 7 days. An expired or unknown link says so; save the contact point again for a new one.
• Saving the contact point again sends a pending address a new confirmation, at most once an hour per address. The console’s Resend confirmation does this.
• Until it’s confirmed, an address gets nothing: the delivery log shows it skipped, “awaiting confirmation”. A contact point with no confirmed address notifies no one.
• The confirmation page allows 30 requests a minute from one IP address. Valid unsubscribe links aren’t limited.

Unsubscribing

The unsubscribe link (and the one-click unsubscribe of mail clients) stops the address receiving the project’s alerts, from every contact point. An unsubscribed address is never asked again automatically: not when a contact point adds it again, not when one is saved. Only an explicit request does:

POST /v1/organizations/<organization ID>/projects/<project ID>/obs/contact-points/<contact point ID>/addresses/<address>/reconfirm

It sends a new confirmation and answers 202 with the address’s status and sentAt. It’s refused with 429 and Retry-After when a confirmation went to the address in the last 7 days, 409 already_confirmed for a confirmed address, and 404 when the contact point doesn’t list it. It needs obs.contactPoints.update.

Webhook

A webhook receives a POST with Content-Type: application/json and Alertmanager’s webhook body (version 4), plus an ezgh object. Anything that accepts Alertmanager webhooks works unchanged.

{
  "version": "4",
  "groupKey": "{}:{alertname=\"API p95 latency\"}",
  "truncatedAlerts": 0,
  "status": "firing",
  "receiver": "PagerBridge webhook",
  "groupLabels": { "alertname": "API p95 latency" },
  "commonLabels": {
    "alertname": "API p95 latency",
    "ezgh_rule": "0199b2c4-5e6f-7a8b-8c9d-0e1f2a3b4c5d",
    "severity": "critical",
    "team": "platform"
  },
  "commonAnnotations": { "description": "Above 500 ms for 5 minutes." },
  "externalURL": "https://console.ezghcloud.com",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "API p95 latency",
        "ezgh_rule": "0199b2c4-5e6f-7a8b-8c9d-0e1f2a3b4c5d",
        "service": "payments/api",
        "severity": "critical",
        "team": "platform"
      },
      "annotations": { "description": "Above 500 ms for 5 minutes.", "summary": "payments/api p95 is 612ms" },
      "startsAt": "2026-10-14T02:00:00Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "https://console.ezghcloud.com/obs/alerting/rules/0199b2c4-5e6f-7a8b-8c9d-0e1f2a3b4c5d?project=0199b2c4-3c4d-7e5f-9a0b-1c2d3e4f5a6b",
      "fingerprint": "3f9a1c0d2b4e6a81"
    },
    {
      "status": "firing",
      "labels": {
        "alertname": "API p95 latency",
        "ezgh_rule": "0199b2c4-5e6f-7a8b-8c9d-0e1f2a3b4c5d",
        "service": "email-worker",
        "severity": "critical",
        "team": "platform"
      },
      "annotations": { "description": "Above 500 ms for 5 minutes.", "summary": "email-worker p95 is 907ms" },
      "startsAt": "2026-10-14T01:31:00Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "https://console.ezghcloud.com/obs/alerting/rules/0199b2c4-5e6f-7a8b-8c9d-0e1f2a3b4c5d?project=0199b2c4-3c4d-7e5f-9a0b-1c2d3e4f5a6b",
      "fingerprint": "a07c55e1d9b3f420"
    }
  ],
  "ezgh": {
    "organization": "0199b2c4-1a2b-7c3d-8e4f-5a6b7c8d9e0f",
    "project": "0199b2c4-3c4d-7e5f-9a0b-1c2d3e4f5a6b",
    "rule": "0199b2c4-5e6f-7a8b-8c9d-0e1f2a3b4c5d",
    "slo": null,
    "tier": null,
    "kind": "firing",
    "notification": "ntf_agm3frd2rn6j3dqpdivtytk6n4",
    "test": false,
    "consoleUrl": "https://console.ezghcloud.com/obs/alerting/rules/0199b2c4-5e6f-7a8b-8c9d-0e1f2a3b4c5d?project=0199b2c4-3c4d-7e5f-9a0b-1c2d3e4f5a6b"
  }
}

• status is resolved only when every alert in it is resolved. An alert’s endsAt is 0001-01-01T00:00:00Z while it’s firing, as in Alertmanager. receiver is the contact point’s name, and groupLabels is the rule’s alertname.
• commonLabels and commonAnnotations are those every alert in the body shares.
• ezgh.organization, ezgh.project and ezgh.rule are the IDs used in API paths. ezgh.kind is firing, noData, repeat, resolved, error or test; ezgh.test is true for Send test. An error notification has no alerts: read the rule’s error from its status (GET alert-rules/<ID>) or the console link.
• A body is at most 1 MiB: past it, alerts are left out from the end and counted in truncatedAlerts, as Alertmanager’s max_alerts does.

Headers

Moving a webhook to another host needs its header values sent again, since they may be credentials for the old host. Sending headers replaces them all.

The secret

Every webhook has a signing secret. Set your own secret (16 to 256 characters), or leave it out when you create the contact point: one is generated and returned once, as webhook.secret in the create’s answer. The console shows it once with Copy; ezgh obs contact-points apply prints it on stderr. Store it then: it can’t be shown again. To rotate it, set a new secret. A contact point changed to a webhook from another type must be given a secret.

Verifying signatures

X-EZGH-Signature is t=<unix seconds>,v1=<signature>, where the signature is the hex HMAC-SHA256 of <t>.<body> (the timestamp, a dot, and the raw request body) keyed with the secret as text. To verify a request:

1. Read the raw body before parsing it: re-serialized JSON doesn’t match.
2. Compute the HMAC of t + . + body with your secret, and compare it with v1 in constant time.
3. Reject timestamps more than 5 minutes from now, so a captured request can’t be replayed later.

import { createHmac, timingSafeEqual } from "node:crypto";

// secret: the contact point's signing secret. header: the X-EZGH-Signature value.
// rawBody: the request body exactly as received (a Buffer), before any JSON parsing.
export function verifyEzghSignature(secret, header, rawBody, now = Date.now() / 1000) {
  const parts = {};
  for (const item of (header ?? "").split(",")) {
    const i = item.indexOf("=");
    if (i > 0) parts[item.slice(0, i).trim()] = item.slice(i + 1).trim();
  }
  if (!/^\d+$/.test(parts.t ?? "") || Math.abs(now - Number(parts.t)) > 300) return false;
  const want = createHmac("sha256", secret).update(`${parts.t}.`).update(rawBody).digest();
  const got = Buffer.from(parts.v1 ?? "", "hex");
  return got.length === want.length && timingSafeEqual(got, want);
}

import hashlib
import hmac
import time


def verify_ezgh_signature(secret: str, header: str, raw_body: bytes, now=None) -> bool:
    """secret: the contact point's signing secret. header: the X-EZGH-Signature value.
    raw_body: the request body exactly as received, before any JSON parsing."""
    parts = dict(item.strip().split("=", 1) for item in (header or "").split(",") if "=" in item)
    ts = parts.get("t", "")
    if not ts.isdigit() or abs((time.time() if now is None else now) - int(ts)) > 300:
        return False
    want = hmac.new(secret.encode(), ts.encode() + b"." + raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(want, parts.get("v1", ""))

import (
	"crypto/hmac"
	"crypto/sha256"
	"encoding/hex"
	"strconv"
	"strings"
	"time"
)

// VerifyEZGHSignature checks X-EZGH-Signature. secret is the contact point's signing secret,
// header the X-EZGH-Signature value, and body the request body exactly as received.
func VerifyEZGHSignature(secret, header string, body []byte, now time.Time) bool {
	var ts, sig string
	for _, item := range strings.Split(header, ",") {
		k, v, _ := strings.Cut(strings.TrimSpace(item), "=")
		switch k {
		case "t":
			ts = v
		case "v1":
			sig = v
		}
	}
	t, err := strconv.ParseInt(ts, 10, 64)
	if err != nil {
		return false
	}
	if age := now.Sub(time.Unix(t, 0)); age > 5*time.Minute || age < -5*time.Minute {
		return false
	}
	got, err := hex.DecodeString(sig)
	if err != nil {
		return false
	}
	mac := hmac.New(sha256.New, []byte(secret))
	mac.Write([]byte(ts + "."))
	mac.Write(body)
	return hmac.Equal(got, mac.Sum(nil))
}

Where webhooks can go

• https only, on port 443 or 8443, with no user name or password in the URL. URLs are up to 2,048 characters.
• The URL must name a host, not an IP address, and the host must resolve to public addresses only. Loopback, private, link-local, shared (CGNAT), multicast, documentation and unspecified addresses, IPv4 and IPv6, are refused at every delivery, after resolving the host. A refused address, or an IP address in the URL, fails the delivery without retrying.
• Redirects aren’t followed: a 3xx answer fails the delivery.
• A delivery times out after 10 seconds. Only the first 4 KiB of your answer is read.

Slack

A Slack contact point posts to a Slack incoming webhook, https://hooks.slack.com/services/…; Slack apps and bot tokens aren’t supported. The message shows the state in color, the rule and project, up to 10 instances with their summary, value and labels (“and N more”), a rule’s error message, and a View in EZGH button.

If Slack answers that the webhook is gone (400, 403, 404 or 410, such as channel_not_found), the delivery fails without retrying: create a new incoming webhook and update the contact point’s URL.

Send test

Send test (POST contact-points/<ID>/test, ezgh obs contact-points test) sends a notification with one sample firing alert (alertname EZGHTest, severity info) through the same path as alerts, and answers 202 with its notificationId. Its attempts appear in the delivery log within a minute. An email contact point sends it only to confirmed addresses. Tests are never silenced.

Delivery and retries

Each notification is delivered to its contact point once, retrying until it succeeds or an hour has passed:

• Retries after 5 seconds, 30 seconds, 2 minutes and 5 minutes, then every 10 minutes, for up to 1 hour after the notification was created. Then it fails.
• A 2xx answer is success. 408, 429, 5xx, timeouts and connection errors are retried; 429 and 503 honor Retry-After within the hour. Any other 3xx or 4xx fails at once.
• An email to several addresses retries only the addresses that didn’t get it.
• A rule’s notifications to one contact point are delivered in order: a resolved notification never arrives before the firing one it resolves.
• Silences are checked again just before each attempt. Instances silenced since the notification was created are left out, and a notification left with none is marked silenced instead of sent.

Delivery is at least once: if your endpoint answers after the timeout, the notification can be sent again. Deduplicate with X-EZGH-Delivery.

The delivery log

Notifications and their attempts are kept for 30 days. The console shows them on a rule’s Notifications tab and on a contact point’s page, each notification expandable to its attempts; ezgh obs alerts notifications lists them.

Each attempt records its time, attempt number, result (ok, retry, failed, skipped, dropped or silenced), the HTTP status or error, the latency, and its target: the email address, or the webhook’s host. Never the URL path, a secret or your answer’s body.

In the console

Open Observability > Alerting > Contact points. The list shows each contact point’s type, target (for email, how many addresses and how many are waiting for confirmation), how many rules use it, and its last delivery. Create contact point asks for the type and its settings; a webhook’s generated secret is shown once. A contact point’s page shows its addresses and their status (Confirmed, Waiting for confirmation, Unsubscribed), the rules that use it, Send test, and its notifications, filtered by status.

From the CLI and the API

ezgh obs contact-points apply -f contact-points/platform.json --project web
OPS_HOOK_SECRET=… ezgh obs contact-points apply -f ops-hook.json --webhook-secret-env OPS_HOOK_SECRET
ezgh obs contact-points list --project web
ezgh obs contact-points get "Platform team" -o json > platform.json
ezgh obs contact-points test "Platform team" --wait
ezgh obs alerts notifications --contact-point "Platform team" --status failed,due
ezgh obs contact-points delete "Platform team"