---
title: "Deliver Trails to Observability"
description: "Deliver your organization's Trails events, as they happen, into a project's Observability as structured logs, and keep them as long as the dataset's log retention."
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.ezghcloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Deliver Trails to Observability

A **delivery** sends your organization's Trails events, from the moment you create it, into one
project's [Observability](/observability) as log records. Trails keeps events for 30 days; the
delivered copy stays as long as the destination dataset's log retention, and you search it, chart
it and alert on it like any other logs.

- **Organization-wide:** a delivery carries every event of the organization, from every project
  and the organization itself, into one project.
- **One environment:** the events go to the dataset of the environment you choose (`production`
  by default), which is [created on first use](/observability#datasets-are-environments) like
  any other.
- **What Trails keeps, or reads too:** a delivery carries the events Trails keeps
  ([What Trails records](/trails#what-trails-records)). With **Include read events**, it also
  carries every read-only API call, most of which Trails itself doesn't keep.
- **From now on:** events from before the delivery was created aren't delivered.

An organization can have up to 5 deliveries.

## Create a delivery

You need `audit.deliveries.create`, and `obs.logs.ingest` on the destination project (see
[Permissions](#permissions)).

### Console

1. In the [console](https://console.ezghcloud.com), open **Trails** and select the
**Deliveries** tab.
2. Select **Create delivery**.
3. Enter a **Name**, choose the **Project**, and set the **Environment** (`production` unless
you change it).
4. Turn on **Include read events** to deliver read-only API calls too.
5. Select **Create delivery**. The delivery's page opens.

The delivery's page shows its status, when it last delivered and how many events were dropped.
**View logs** opens the environment's logs in Observability, filtered to Trails' records.
**Edit** changes the name and **Include read events**; **Pause** and **Resume** stop and restart
delivery; **Delete** removes it.
### CLI

`ezgh trails deliveries create` delivers to the project given by `--project` (or `EZGH_PROJECT`,
or the profile's project):

```sh
ezgh trails deliveries create "Audit archive" --project security --environment audit
ezgh trails deliveries create "Everything" --project security --include-reads
```

| Command | Does |
| --- | --- |
| `ezgh trails deliveries list` | Lists deliveries: name, destination, reads and status. |
| `ezgh trails deliveries get <delivery>` | Shows a delivery by ID or slug (`tdl_…`), with its status, last delivery, last error and dropped events. |
| `ezgh trails deliveries create <name>` | Creates a delivery. `--environment` (default `production`), `--include-reads`, and `--enabled=false` to create it paused. |
| `ezgh trails deliveries update <delivery>` | Changes only the flags given: `--name`, `--include-reads=true\|false`, and `--enabled=false` to pause or `--enabled=true` to resume. |
| `ezgh trails deliveries delete <delivery>` | Deletes a delivery. It asks first; `--yes` confirms. |

```
$ ezgh trails deliveries list
NAME            DESTINATION             READS   STATUS   ID
Everything      security · production   yes     active   0199a3f0-0000-7000-8000-0000000d0002
Audit archive   security · audit        no      active   0199a3f0-0000-7000-8000-0000000d0001
```
### API

| Operation | Request | Needs |
| --- | --- | --- |
| [CreateTrailDelivery](/trails-api/deliveries/CreateTrailDelivery/) | `POST /v1/organizations/<organization ID>/deliveries` | `audit.deliveries.create`, and `obs.logs.ingest` on the project |
| [ListTrailDeliveries](/trails-api/deliveries/ListTrailDeliveries/) | `GET …/deliveries` | `audit.deliveries.list` |
| [GetTrailDelivery](/trails-api/deliveries/GetTrailDelivery/) | `GET …/deliveries/<delivery ID or slug>` | `audit.deliveries.get` |
| [UpdateTrailDelivery](/trails-api/deliveries/UpdateTrailDelivery/) | `PATCH …/deliveries/<delivery ID or slug>` | `audit.deliveries.update`, and `obs.logs.ingest` on the project to turn `enabled` or `includeReads` on |
| [DeleteTrailDelivery](/trails-api/deliveries/DeleteTrailDelivery/) | `DELETE …/deliveries/<delivery ID or slug>` | `audit.deliveries.delete` |

```sh
curl -H "Authorization: Bearer $EZGH_API_KEY" -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"name":"Audit archive","projectId":"'"$PROJECT_ID"'","environment":"audit","includeReads":false}' \
  "https://trails.ezghcloud.com/v1/organizations/$ORG_ID/deliveries"
```

```json
{
  "id": "0199a3f0-0000-7000-8000-0000000d0001",
  "slug": "tdl_4k2m9x0a7qpe",
  "resourceName": "ezgh:us-west-1:org_k3f9a0x2m7qp:tdl_4k2m9x0a7qpe",
  "organizationId": "0199a3c2-5b1e-7d40-9f3a-2c8e6b1d4f70",
  "region": "us-west-1",
  "name": "Audit archive",
  "projectId": "0199a3c2-6c2f-7e51-8a4b-3d9f7c2e5a81",
  "environment": "audit",
  "includeReads": false,
  "enabled": true,
  "status": "active",
  "lastDeliveredAt": null,
  "lastError": null,
  "lastErrorAt": null,
  "droppedEvents": 0,
  "createdAt": "2026-10-14T09:00:00.000Z",
  "updatedAt": "2026-10-14T09:00:00.000Z"
}
```

- `CreateTrailDelivery` answers `201` with a `Location` header. It takes an `Idempotency-Key`:
a retry with the same key within 24 hours answers the first `201` again, with
`Idempotent-Replayed: true`.
- `name` is 1 to 100 characters. `environment` is trimmed and lowercased, then must match
`^[a-z0-9][a-z0-9._-]{0,62}$`. `includeReads` defaults to `false` and `enabled` to `true`.
- `projectId` and `environment` can't change: a `PATCH` naming them is `400 invalid_request`.
Create another delivery instead.
- A project that isn't a live project of the organization is `400 invalid_request`.
- Without `obs.logs.ingest` on the project, the answer is `403 access_denied`, naming it.
- The sixth delivery is `409 quota_exceeded`, with the quota
`audit.deliveries.perOrganization` (an allocation of 5) and how many are in use.
- The list is newest first, 20 a page by default and at most 100 (`limit`), with
`nextCursor` for the next page (`cursor`).
- `DeleteTrailDelivery` answers `204`.

## Read events and billing

Delivered records are ordinary Observability logs in the destination project: their bytes and
storage count in that project's [Observability meters](/observability/limits) and go through its
rate limits and [transform rules](/observability/transforms). Trails itself adds no charge.

**Include read events** delivers every read-only API call as well, such as reading a project or
listing bots. Reads are most API calls; they're billed as Observability log ingest in the
destination project like the rest.

## The record

Each event becomes one log record. Its **body is the event's JSON exactly**, as
[the Trails API returns it](/trails/events), so anything that reads Trails events can read the
delivered copy.

**Resource attributes**

| Key | Value |
| --- | --- |
| `service.name` | `ezgh.trails` |
| `ezgh.metadata.application` | `trails` |
| `ezgh.metadata.environment` | The delivery's environment |
| `ezgh.metadata.project` | The event's project ID; absent for organization-level events |

**Record**

| Field | Value |
| --- | --- |
| Time | The event's `eventTime` |
| Observed time | When Trails received the event |
| Severity | `WARN` when the event has an `errorCode`, else `INFO` |
| Body | The event's JSON, as a string |
| Scope | `trails.ezghcloud.com/delivery` |

**Record attributes** copy the event's main fields, for search. A field the event doesn't have
is left out, never sent empty.

| Key | From |
| --- | --- |
| `trails.eventId`, `trails.eventName`, `trails.eventSource`, `trails.eventType` | The same fields |
| `trails.readOnly` | `"true"` or `"false"` |
| `trails.errorCode` | `errorCode` |
| `trails.userIdentity.type`, `trails.userIdentity.principalId`, `trails.userIdentity.userName`, `trails.userIdentity.accessKeyId`, `trails.userIdentity.invokedBy` | The same fields of `userIdentity` |
| `trails.sourceIpAddress`, `trails.requestId` | The same fields |
| `trails.organizationId` | `organizationId` |
| `trails.resources` | The resources' names, comma-separated |
| `trails.deliveryId` | The delivery |

An example record, as OTLP JSON (the body shortened):

```json
{
  "timeUnixNano": "1791964800123000000",
  "observedTimeUnixNano": "1791964801250000000",
  "severityNumber": 9,
  "severityText": "INFO",
  "body": { "stringValue": "{\"additionalEventData\":null,\"errorCode\":null,…,\"eventName\":\"CreateDataset\",…}" },
  "attributes": [
    { "key": "trails.eventId", "value": { "stringValue": "0199a3f2-7c1d-7a4b-9e2f-1b3c4d5e6f70" } },
    { "key": "trails.eventName", "value": { "stringValue": "CreateDataset" } },
    { "key": "trails.eventSource", "value": { "stringValue": "obs.ezghcloud.com" } },
    { "key": "trails.eventType", "value": { "stringValue": "ApiCall" } },
    { "key": "trails.readOnly", "value": { "stringValue": "false" } },
    { "key": "trails.userIdentity.type", "value": { "stringValue": "User" } },
    { "key": "trails.userIdentity.principalId", "value": { "stringValue": "user_7f3k2m" } },
    { "key": "trails.userIdentity.userName", "value": { "stringValue": "ada@example.com" } },
    { "key": "trails.sourceIpAddress", "value": { "stringValue": "203.0.113.7" } },
    { "key": "trails.requestId", "value": { "stringValue": "req_1" } },
    { "key": "trails.organizationId", "value": { "stringValue": "01990000-0000-7000-8000-000000000001" } },
    { "key": "trails.resources", "value": { "stringValue": "ezgh:us-west-1:org_k3f9a0x2m7qp:prj_x:ds_web" } },
    { "key": "trails.deliveryId", "value": { "stringValue": "01990000-0000-7000-8000-0000000000d1" } }
  ]
}
```

Your project's transform rules apply to delivered records like any others. Trails remains the
record of what happened for its 30 days.

## Search the logs

In the console, open **Observability**, then **Logs**, choose the delivery's environment, and
filter on the service `ezgh.trails`; a delivery's **View logs** opens exactly that. Add
attribute filters such as `trails.eventName`, `trails.userIdentity.userName` or
`trails.userIdentity.accessKeyId`.

With the CLI:

```sh
# Every bot deleted in the last day.
ezgh obs search --project security --environment audit --from 1d \
  --service ezgh.trails --attr trails.eventName=DeleteBot

# Refused calls.
ezgh obs search --project security --environment audit --from 7d \
  --service ezgh.trails --attr trails.errorCode=access_denied
```

## Status and dropped events

| Status | Means |
| --- | --- |
| `active` | Delivering. |
| `paused` | Turned off (`enabled: false`). Nothing is delivered until it's resumed; events from while it was paused aren't delivered. |
| `failing` | The last attempt to deliver failed after the last success. `lastError` says why. |

A delivery that can't deliver (Observability unavailable, or the project's limits refusing it)
keeps its events and retries, with increasing waits, for up to **3 days**. Events still
undelivered after 3 days are dropped. `droppedEvents` counts every event that was never
delivered: those dropped after 3 days, and records Observability rejected, such as one whose
`eventTime` falls outside the dataset's accepted window.

Delivery is **at least once**: after a retry, a record can arrive twice. Use `trails.eventId` to
tell repeats apart.

Deleting the destination project, or the organization, deletes its deliveries. Deleting a delivery
drops the events it hasn't delivered; what it delivered stays in Observability.

## Permissions

| Action | Allows | In `AdministratorAccess` and `TrailsFullAccess` | In `ReadOnlyAccess` |
| --- | --- | --- | --- |
| `audit.deliveries.list` | List the organization's deliveries | Yes | Yes |
| `audit.deliveries.get` | View a delivery and its status | Yes | Yes |
| `audit.deliveries.create` | Create a delivery | Yes | No |
| `audit.deliveries.update` | Rename, pause or resume a delivery, or turn read events on or off | Yes | No |
| `audit.deliveries.delete` | Delete a delivery | Yes | No |

**Creating a delivery, or turning `enabled` or `includeReads` on,** sends events into the
destination project's Observability, so it also needs `obs.logs.ingest` on that project, as
creating an [ingest key](/observability/ingest-keys) does. Reading the delivered logs needs the
usual Observability actions, such as `obs.logs.search`.

Creating, changing and deleting a delivery are recorded in Trails as `CreateTrailDelivery`,
`UpdateTrailDelivery` and `DeleteTrailDelivery`, with the delivery (`TrailDelivery`) in
`resources`. A delivery deleted with its project or organization is a service event by
`organizations.ezghcloud.com`.

Source: https://docs.ezghcloud.com/trails/deliveries/index.mdx
