---
title: "Contact points"
description: "Send alert notifications to confirmed email addresses, signed Alertmanager-compatible webhooks and Slack, with test notifications, retries and a delivery log."
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.ezghcloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Contact points

A contact point is where a rule's notifications go. Each is one of:

| Type | Settings | Rules |
| --- | --- | --- |
| `email` | `addresses`: 1 to 10 | Each address confirms once for the project before it receives alerts |
| `webhook` | `url`, `secret`, `headers` (up to 5) | `https` on port 443 or 8443, a host name with public addresses, no credentials in the URL |
| `slack` | `url` | A Slack incoming webhook, `https://hooks.slack.com/services/…` |

```json
{ "name": "Platform team", "type": "email", "email": { "addresses": ["ops@example.com", "oncall@example.com"] } }
```

```json
{ "name": "PagerBridge webhook", "type": "webhook", "webhook": { "url": "https://hooks.example.com/ezgh", "headers": { "Authorization": "Bearer …" } } }
```

```json
{ "name": "#platform-alerts", "type": "slack", "slack": { "url": "https://hooks.slack.com/services/T0001/B0002/abcdef" } }
```

- `name` is 1 to 100 characters. A rule lists up to 10 contact points by ID.
- **Write-only values** are never returned: a webhook's secret and header values, and the Slack
  URL. Answers show the webhook's URL and host, `hasSecret`, the header names, and the Slack URL
  masked as `hooks.slack.com/services/T…/…`. A change that leaves them out keeps the stored ones,
  so a file in Git never has to carry them. They're stored encrypted.
- A contact point that rules notify can't be deleted (`409 contact_point_in_use`, listing them in
  the error's `details`): take it out of them first. Notifications to a deleted contact
  point that haven't been sent are dropped.
- Changes apply to the next delivery, including notifications already waiting to be retried.

## Email

Alert email comes from **`EZGH Cloud Alerts <alerts@alerts.ezghcloud.com>`**. Add the address to
your allow-list, or to your mail filters, so it isn't treated as spam.

- **Subject:** `[FIRING:2] API p95 latency (<project>)`, `[RESOLVED] …`, `[NO DATA] …`,
  `[ERROR] …` or `[TEST] Test notification`.
- **Body** (plain text and HTML): the state, rule and project, up to 20 instances with their
  summary, value and labels ("and N more"), a rule's error message, and a link to the rule in the
  console.
- Each address gets its own message, with a one-click **unsubscribe** link and the
  `List-Unsubscribe` and `List-Unsubscribe-Post` headers, so mail clients offer to unsubscribe.

### Confirmation

Any address can receive alerts, shared inboxes and lists included, once it **opts in for the
project**. Every address confirms by email, organization members' too.

1. Saving a contact point with an address the project hasn't confirmed sends it a confirmation
   email, "Confirm alert emails for &lt;organization&gt;/&lt;project&gt;", saying who added it.
2. Its **Confirm** link opens a page on `alerts.ezghcloud.com` with a button. The address is
   confirmed when the button is pressed, not when the link is opened, so mail scanners that open
   links don't confirm it.
3. The address then receives alerts from every contact point in the project that lists it.

- The link works for **7 days**. An expired or unknown link says so; save the contact point again
  for a new one.
- Saving the contact point again sends a pending address a new confirmation, at most once an hour
  per address. The console's **Resend confirmation** does this.
- Until it's confirmed, an address gets nothing: the delivery log shows it skipped, "awaiting
  confirmation". A contact point with no confirmed address notifies no one.
- The confirmation page allows 30 requests a minute from one IP address. Valid unsubscribe links
  aren't limited.

### Unsubscribing

The unsubscribe link (and the one-click unsubscribe of mail clients) stops the address receiving
the project's alerts, from every contact point. An unsubscribed address is **never asked again
automatically**: not when a contact point adds it again, not when one is saved. Only an explicit
request does:

```text
POST /v1/organizations/<organization ID>/projects/<project ID>/obs/contact-points/<contact point ID>/addresses/<address>/reconfirm
```

It sends a new confirmation and answers `202` with the address's `status` and `sentAt`. It's
refused with `429` and `Retry-After` when a confirmation went to the address in the last 7 days,
`409 already_confirmed` for a confirmed address, and `404` when the contact point doesn't list it.
It needs `obs.contactPoints.update`.

## Webhook

A webhook receives a `POST` with `Content-Type: application/json` and **Alertmanager's webhook
body (version 4)**, plus an `ezgh` object. Anything that accepts Alertmanager webhooks works
unchanged.

```json
{
  "version": "4",
  "groupKey": "{}:{alertname=\"API p95 latency\"}",
  "truncatedAlerts": 0,
  "status": "firing",
  "receiver": "PagerBridge webhook",
  "groupLabels": { "alertname": "API p95 latency" },
  "commonLabels": {
    "alertname": "API p95 latency",
    "ezgh_rule": "0199b2c4-5e6f-7a8b-8c9d-0e1f2a3b4c5d",
    "severity": "critical",
    "team": "platform"
  },
  "commonAnnotations": { "description": "Above 500 ms for 5 minutes." },
  "externalURL": "https://console.ezghcloud.com",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "API p95 latency",
        "ezgh_rule": "0199b2c4-5e6f-7a8b-8c9d-0e1f2a3b4c5d",
        "service": "payments/api",
        "severity": "critical",
        "team": "platform"
      },
      "annotations": { "description": "Above 500 ms for 5 minutes.", "summary": "payments/api p95 is 612ms" },
      "startsAt": "2026-10-14T02:00:00Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "https://console.ezghcloud.com/obs/alerting/rules/0199b2c4-5e6f-7a8b-8c9d-0e1f2a3b4c5d?project=0199b2c4-3c4d-7e5f-9a0b-1c2d3e4f5a6b",
      "fingerprint": "3f9a1c0d2b4e6a81"
    },
    {
      "status": "firing",
      "labels": {
        "alertname": "API p95 latency",
        "ezgh_rule": "0199b2c4-5e6f-7a8b-8c9d-0e1f2a3b4c5d",
        "service": "email-worker",
        "severity": "critical",
        "team": "platform"
      },
      "annotations": { "description": "Above 500 ms for 5 minutes.", "summary": "email-worker p95 is 907ms" },
      "startsAt": "2026-10-14T01:31:00Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "https://console.ezghcloud.com/obs/alerting/rules/0199b2c4-5e6f-7a8b-8c9d-0e1f2a3b4c5d?project=0199b2c4-3c4d-7e5f-9a0b-1c2d3e4f5a6b",
      "fingerprint": "a07c55e1d9b3f420"
    }
  ],
  "ezgh": {
    "organization": "0199b2c4-1a2b-7c3d-8e4f-5a6b7c8d9e0f",
    "project": "0199b2c4-3c4d-7e5f-9a0b-1c2d3e4f5a6b",
    "rule": "0199b2c4-5e6f-7a8b-8c9d-0e1f2a3b4c5d",
    "slo": null,
    "tier": null,
    "kind": "firing",
    "notification": "ntf_agm3frd2rn6j3dqpdivtytk6n4",
    "test": false,
    "consoleUrl": "https://console.ezghcloud.com/obs/alerting/rules/0199b2c4-5e6f-7a8b-8c9d-0e1f2a3b4c5d?project=0199b2c4-3c4d-7e5f-9a0b-1c2d3e4f5a6b"
  }
}
```

- `status` is `resolved` only when every alert in it is resolved. An alert's `endsAt` is
  `0001-01-01T00:00:00Z` while it's firing, as in Alertmanager. `receiver` is the contact point's
  name, and `groupLabels` is the rule's `alertname`.
- `commonLabels` and `commonAnnotations` are those every alert in the body shares.
- `ezgh.organization`, `ezgh.project` and `ezgh.rule` are the IDs used in API paths.
  `ezgh.kind` is `firing`, `noData`, `repeat`, `resolved`, `error` or `test`; `ezgh.test` is
  `true` for **Send test**. An `error` notification has no alerts: read the rule's error from its
  status (`GET alert-rules/<ID>`) or the console link.
- A body is at most 1 MiB: past it, alerts are left out from the end and counted in
  `truncatedAlerts`, as Alertmanager's `max_alerts` does.

### Headers

| Header | Value |
| --- | --- |
| `X-EZGH-Signature` | `t=<unix seconds>,v1=<hex HMAC-SHA256>`; see [Verifying signatures](#verifying-signatures) |
| `X-EZGH-Delivery` | The notification's ID, the same as `ezgh.notification`. It's the same on every retry: use it to ignore duplicates |
| Your headers | Up to 5, such as `Authorization`. Values are up to 1,024 characters and write-only. `Content-Type`, `Host`, `Content-Length`, `Transfer-Encoding`, `Connection`, `User-Agent` and `X-EZGH-*` are set by EZGH |

Moving a webhook to another host needs its header values sent again, since they may be credentials
for the old host. Sending `headers` replaces them all.

### The secret

Every webhook has a signing secret. Set your own `secret` (16 to 256 characters), or leave it out
when you create the contact point: one is generated and **returned once**, as `webhook.secret` in
the create's answer. The console shows it once with **Copy**; `ezgh obs contact-points apply`
prints it on stderr. Store it then: it can't be shown again. To rotate it, set a new `secret`. A
contact point changed to a webhook from another type must be given a secret.

### Verifying signatures

`X-EZGH-Signature` is `t=<unix seconds>,v1=<signature>`, where the signature is the hex
HMAC-SHA256 of `<t>.<body>` (the timestamp, a dot, and the raw request body) keyed with the
secret as text. To verify a request:

1. Read the raw body **before parsing it**: re-serialized JSON doesn't match.
2. Compute the HMAC of `t` + `.` + body with your secret, and compare it with `v1` in constant time.
3. Reject timestamps more than 5 minutes from now, so a captured request can't be replayed later.

### Node.js

```js
import { createHmac, timingSafeEqual } from "node:crypto";

// secret: the contact point's signing secret. header: the X-EZGH-Signature value.
// rawBody: the request body exactly as received (a Buffer), before any JSON parsing.
export function verifyEzghSignature(secret, header, rawBody, now = Date.now() / 1000) {
  const parts = {};
  for (const item of (header ?? "").split(",")) {
    const i = item.indexOf("=");
    if (i > 0) parts[item.slice(0, i).trim()] = item.slice(i + 1).trim();
  }
  if (!/^\d+$/.test(parts.t ?? "") || Math.abs(now - Number(parts.t)) > 300) return false;
  const want = createHmac("sha256", secret).update(`${parts.t}.`).update(rawBody).digest();
  const got = Buffer.from(parts.v1 ?? "", "hex");
  return got.length === want.length && timingSafeEqual(got, want);
}
```

With Express, read the body raw: `app.post("/ezgh", express.raw({ type: "application/json" }), …)`,
then `verifyEzghSignature(secret, req.get("X-EZGH-Signature"), req.body)`.
### Python

```python
import hashlib
import hmac
import time


def verify_ezgh_signature(secret: str, header: str, raw_body: bytes, now=None) -> bool:
    """secret: the contact point's signing secret. header: the X-EZGH-Signature value.
    raw_body: the request body exactly as received, before any JSON parsing."""
    parts = dict(item.strip().split("=", 1) for item in (header or "").split(",") if "=" in item)
    ts = parts.get("t", "")
    if not ts.isdigit() or abs((time.time() if now is None else now) - int(ts)) > 300:
        return False
    want = hmac.new(secret.encode(), ts.encode() + b"." + raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(want, parts.get("v1", ""))
```

With Flask, `verify_ezgh_signature(secret, request.headers.get("X-EZGH-Signature"), request.get_data())`.
### Go

```go
import (
	"crypto/hmac"
	"crypto/sha256"
	"encoding/hex"
	"strconv"
	"strings"
	"time"
)

// VerifyEZGHSignature checks X-EZGH-Signature. secret is the contact point's signing secret,
// header the X-EZGH-Signature value, and body the request body exactly as received.
func VerifyEZGHSignature(secret, header string, body []byte, now time.Time) bool {
	var ts, sig string
	for _, item := range strings.Split(header, ",") {
		k, v, _ := strings.Cut(strings.TrimSpace(item), "=")
		switch k {
		case "t":
			ts = v
		case "v1":
			sig = v
		}
	}
	t, err := strconv.ParseInt(ts, 10, 64)
	if err != nil {
		return false
	}
	if age := now.Sub(time.Unix(t, 0)); age > 5*time.Minute || age < -5*time.Minute {
		return false
	}
	got, err := hex.DecodeString(sig)
	if err != nil {
		return false
	}
	mac := hmac.New(sha256.New, []byte(secret))
	mac.Write([]byte(ts + "."))
	mac.Write(body)
	return hmac.Equal(got, mac.Sum(nil))
}
```

In an `http.Handler`, `body, _ := io.ReadAll(r.Body)`, then
`VerifyEZGHSignature(secret, r.Header.Get("X-EZGH-Signature"), body, time.Now())`.

### Where webhooks can go

- `https` only, on port 443 or 8443, with no user name or password in the URL. URLs are up to
  2,048 characters.
- The URL must name a host, not an IP address, and the host must resolve to public addresses
  only. Loopback, private, link-local, shared (CGNAT), multicast, documentation and unspecified
  addresses, IPv4 and IPv6, are refused at every delivery, after resolving the host. A refused
  address, or an IP address in the URL, fails the delivery without retrying.
- Redirects aren't followed: a `3xx` answer fails the delivery.
- A delivery times out after 10 seconds. Only the first 4 KiB of your answer is read.

## Slack

A Slack contact point posts to a Slack [incoming webhook](https://api.slack.com/messaging/webhooks),
`https://hooks.slack.com/services/…`; Slack apps and bot tokens aren't supported. The message
shows the state in color, the rule and project, up to 10 instances with their summary, value and
labels ("and N more"), a rule's error message, and a **View in EZGH** button.

If Slack answers that the webhook is gone (`400`, `403`, `404` or `410`, such as
`channel_not_found`), the delivery fails without retrying: create a new incoming webhook and update
the contact point's URL.

## Send test

**Send test** (`POST contact-points/<ID>/test`, `ezgh obs contact-points test`) sends a
notification with one sample firing alert (`alertname` `EZGHTest`, `severity` `info`) through the
same path as alerts, and answers `202` with its `notificationId`. Its attempts appear in the
delivery log within a minute. An email contact point sends it only to confirmed addresses. Tests
are never silenced.

## Delivery and retries

Each notification is delivered to its contact point once, retrying until it succeeds or an hour
has passed:

- **Retries** after 5 seconds, 30 seconds, 2 minutes and 5 minutes, then every 10 minutes, for
  up to **1 hour** after the notification was created. Then it fails.
- A `2xx` answer is success. `408`, `429`, `5xx`, timeouts and connection errors are retried;
  `429` and `503` honor `Retry-After` within the hour. Any other `3xx` or `4xx` fails at once.
- An email to several addresses retries only the addresses that didn't get it.
- A rule's notifications to one contact point are delivered **in order**: a resolved
  notification never arrives before the firing one it resolves.
- Silences are checked again just before each attempt. Instances silenced since the notification
  was created are left out, and a notification left with none is marked silenced instead of sent.

Delivery is at least once: if your endpoint answers after the timeout, the notification can be
sent again. Deduplicate with `X-EZGH-Delivery`.

### The delivery log

Notifications and their attempts are kept for **30 days**. The console shows them on a rule's
**Notifications** tab and on a contact point's page, each notification expandable to its
attempts; `ezgh obs alerts notifications` lists them.

| Notification status | Meaning |
| --- | --- |
| Queued or **Retrying** (`due`) | Waiting for its first attempt or its next retry |
| **Sent** (`sent`) | Delivered. An email notification whose addresses were all skipped (not confirmed or unsubscribed) is also sent: the attempts say why |
| **Failed** (`failed`) | An attempt failed without retry, or the hour ran out |
| **Dropped** (`dropped`) | Not sent: the contact point was deleted, or a confirmation's address was confirmed, unsubscribed or removed in the meantime |
| **Silenced** (`silenced`) | Every instance in it was silenced before it could be sent |

Each attempt records its time, attempt number, result (`ok`, `retry`, `failed`, `skipped`,
`dropped` or `silenced`), the HTTP status or error, the latency, and its target: the email
address, or the webhook's host. Never the URL path, a secret or your answer's body.

## In the console

Open **Observability** > **Alerting** > **Contact points**. The list shows each contact point's
type, target (for email, how many addresses and how many are waiting for confirmation), how many
rules use it, and its last delivery. **Create contact point** asks for the type
and its settings; a webhook's generated secret is shown once. A contact point's page shows its
addresses and their status (**Confirmed**, **Waiting for confirmation**, **Unsubscribed**), the
rules that use it, **Send test**, and its notifications, filtered by status.

## From the CLI and the API

### CLI

```sh
ezgh obs contact-points apply -f contact-points/platform.json --project web
OPS_HOOK_SECRET=… ezgh obs contact-points apply -f ops-hook.json --webhook-secret-env OPS_HOOK_SECRET
ezgh obs contact-points list --project web
ezgh obs contact-points get "Platform team" -o json > platform.json
ezgh obs contact-points test "Platform team" --wait
ezgh obs alerts notifications --contact-point "Platform team" --status failed,due
ezgh obs contact-points delete "Platform team"
```

- `apply` updates the contact point the file's `id` or `slug` names, else the one with the file's
`name` when exactly one has it, and creates one when there's none. `--webhook-secret-env` and
`--slack-url-env` read the write-only values from environment variables instead of the file.
- `test --wait` waits up to 70 seconds for the delivery's result. It exits `1` if the delivery
failed, if every address was skipped (not confirmed or unsubscribed), or if there's no result
after 70 seconds.

[`ezgh obs contact-points`](/cli/reference/obs-contact-points) and [`ezgh obs alerts notifications`](/cli/reference/obs-alerts-notifications) list every flag.
### API

Under `/v1/organizations/<organization ID>/projects/<project ID>/obs/`:

| Operation | Request |
| --- | --- |
| [List contact points](/obs-api/ListContactPoints/) | `GET contact-points` |
| [Create a contact point](/obs-api/CreateContactPoint/) | `POST contact-points` |
| [Get a contact point](/obs-api/GetContactPoint/) | `GET contact-points/<ID>` |
| [Change a contact point](/obs-api/UpdateContactPoint/) | `PATCH contact-points/<ID>` |
| [Delete a contact point](/obs-api/DeleteContactPoint/) | `DELETE contact-points/<ID>` |
| [Send a test](/obs-api/TestContactPoint/) | `POST contact-points/<ID>/test` |
| [Ask an address to confirm again](/obs-api/ReconfirmContactPointAddress/) | `POST contact-points/<ID>/addresses/<address>/reconfirm` |
| [Notifications, with their attempts](/obs-api/ListAlertNotifications/) | `GET alerts/notifications` |
| [Delivery attempts](/obs-api/ListAlertDeliveries/) | `GET alerts/deliveries` |

- A contact point's answer has its `email.addresses` with each one's `status` (`pending`,
`confirmed` or `unsubscribed`) and `sentAt`; or `webhook` `url`, `host`, `hasSecret` and
`headerNames`; or the masked `slack.url`. It also has `usedBy` (`rules` and `slos`: how many notify it) and
`lastDelivery`.
- A `PATCH` that changes the `webhook` or `slack` object merges it field by field, so write-only
values you leave out are kept.
- `GET alerts/notifications` filters by `contactPoint`, `owner` (`rule:<rule ID>`), `status` and
`kind`, newest first. Each notification has its latest 20 attempts, and `attemptsTotal` when
there are more. `GET alerts/deliveries` lists attempts, filtered by `contactPoint`, `rule`,
`notification` and `result`.

Source: https://docs.ezghcloud.com/observability/alerting/contact-points/index.mdx