Skip to content

Deliver Trails to Observability

Deliver your organization's Trails events, as they happen, into a project's Observability as structured logs, and keep them as long as the dataset's log retention.

Updated View as Markdown

A delivery sends your organization’s Trails events, from the moment you create it, into one project’s Observability as log records. Trails keeps events for 30 days; the delivered copy stays as long as the destination dataset’s log retention, and you search it, chart it and alert on it like any other logs.

  • Organization-wide: a delivery carries every event of the organization, from every project and the organization itself, into one project.
  • One environment: the events go to the dataset of the environment you choose (production by default), which is created on first use like any other.
  • What Trails keeps, or reads too: a delivery carries the events Trails keeps (What Trails records). With Include read events, it also carries every read-only API call, most of which Trails itself doesn’t keep.
  • From now on: events from before the delivery was created aren’t delivered.

An organization can have up to 5 deliveries.

Create a delivery

You need audit.deliveries.create, and obs.logs.ingest on the destination project (see Permissions).

  1. In the console, open Trails and select the Deliveries tab.
  2. Select Create delivery.
  3. Enter a Name, choose the Project, and set the Environment (production unless you change it).
  4. Turn on Include read events to deliver read-only API calls too.
  5. Select Create delivery. The delivery’s page opens.

The delivery’s page shows its status, when it last delivered and how many events were dropped. View logs opens the environment’s logs in Observability, filtered to Trails’ records. Edit changes the name and Include read events; Pause and Resume stop and restart delivery; Delete removes it.

ezgh trails deliveries create delivers to the project given by --project (or EZGH_PROJECT, or the profile’s project):

ezgh trails deliveries create "Audit archive" --project security --environment audit
ezgh trails deliveries create "Everything" --project security --include-reads
Command Does
ezgh trails deliveries list Lists deliveries: name, destination, reads and status.
ezgh trails deliveries get <delivery> Shows a delivery by ID or slug (tdl_…), with its status, last delivery, last error and dropped events.
ezgh trails deliveries create <name> Creates a delivery. --environment (default production), --include-reads, and --enabled=false to create it paused.
ezgh trails deliveries update <delivery> Changes only the flags given: --name, --include-reads=true|false, and --enabled=false to pause or --enabled=true to resume.
ezgh trails deliveries delete <delivery> Deletes a delivery. It asks first; --yes confirms.
$ ezgh trails deliveries list
NAME            DESTINATION             READS   STATUS   ID
Everything      security · production   yes     active   0199a3f0-0000-7000-8000-0000000d0002
Audit archive   security · audit        no      active   0199a3f0-0000-7000-8000-0000000d0001
Operation Request Needs
CreateTrailDelivery POST /v1/organizations/<organization ID>/deliveries audit.deliveries.create, and obs.logs.ingest on the project
ListTrailDeliveries GET …/deliveries audit.deliveries.list
GetTrailDelivery GET …/deliveries/<delivery ID or slug> audit.deliveries.get
UpdateTrailDelivery PATCH …/deliveries/<delivery ID or slug> audit.deliveries.update, and obs.logs.ingest on the project to turn enabled or includeReads on
DeleteTrailDelivery DELETE …/deliveries/<delivery ID or slug> audit.deliveries.delete
curl -H "Authorization: Bearer $EZGH_API_KEY" -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"name":"Audit archive","projectId":"'"$PROJECT_ID"'","environment":"audit","includeReads":false}' \
  "https://trails.ezghcloud.com/v1/organizations/$ORG_ID/deliveries"
{
  "id": "0199a3f0-0000-7000-8000-0000000d0001",
  "slug": "tdl_4k2m9x0a7qpe",
  "resourceName": "ezgh:us-west-1:org_k3f9a0x2m7qp:tdl_4k2m9x0a7qpe",
  "organizationId": "0199a3c2-5b1e-7d40-9f3a-2c8e6b1d4f70",
  "region": "us-west-1",
  "name": "Audit archive",
  "projectId": "0199a3c2-6c2f-7e51-8a4b-3d9f7c2e5a81",
  "environment": "audit",
  "includeReads": false,
  "enabled": true,
  "status": "active",
  "lastDeliveredAt": null,
  "lastError": null,
  "lastErrorAt": null,
  "droppedEvents": 0,
  "createdAt": "2026-10-14T09:00:00.000Z",
  "updatedAt": "2026-10-14T09:00:00.000Z"
}
  • CreateTrailDelivery answers 201 with a Location header. It takes an Idempotency-Key: a retry with the same key within 24 hours answers the first 201 again, with Idempotent-Replayed: true.
  • name is 1 to 100 characters. environment is trimmed and lowercased, then must match ^[a-z0-9][a-z0-9._-]{0,62}$. includeReads defaults to false and enabled to true.
  • projectId and environment can’t change: a PATCH naming them is 400 invalid_request. Create another delivery instead.
  • A project that isn’t a live project of the organization is 400 invalid_request.
  • Without obs.logs.ingest on the project, the answer is 403 access_denied, naming it.
  • The sixth delivery is 409 quota_exceeded, with the quota audit.deliveries.perOrganization (an allocation of 5) and how many are in use.
  • The list is newest first, 20 a page by default and at most 100 (limit), with nextCursor for the next page (cursor).
  • DeleteTrailDelivery answers 204.

Read events and billing

Delivered records are ordinary Observability logs in the destination project: their bytes and storage count in that project’s Observability meters and go through its rate limits and transform rules. Trails itself adds no charge.

Include read events delivers every read-only API call as well, such as reading a project or listing bots. Reads are most API calls; they’re billed as Observability log ingest in the destination project like the rest.

The record

Each event becomes one log record. Its body is the event’s JSON exactly, as the Trails API returns it, so anything that reads Trails events can read the delivered copy.

Resource attributes

Key Value
service.name ezgh.trails
ezgh.metadata.application trails
ezgh.metadata.environment The delivery’s environment
ezgh.metadata.project The event’s project ID; absent for organization-level events

Record

Field Value
Time The event’s eventTime
Observed time When Trails received the event
Severity WARN when the event has an errorCode, else INFO
Body The event’s JSON, as a string
Scope trails.ezghcloud.com/delivery

Record attributes copy the event’s main fields, for search. A field the event doesn’t have is left out, never sent empty.

Key From
trails.eventId, trails.eventName, trails.eventSource, trails.eventType The same fields
trails.readOnly "true" or "false"
trails.errorCode errorCode
trails.userIdentity.type, trails.userIdentity.principalId, trails.userIdentity.userName, trails.userIdentity.accessKeyId, trails.userIdentity.invokedBy The same fields of userIdentity
trails.sourceIpAddress, trails.requestId The same fields
trails.organizationId organizationId
trails.resources The resources’ names, comma-separated
trails.deliveryId The delivery

An example record, as OTLP JSON (the body shortened):

{
  "timeUnixNano": "1791964800123000000",
  "observedTimeUnixNano": "1791964801250000000",
  "severityNumber": 9,
  "severityText": "INFO",
  "body": { "stringValue": "{\"additionalEventData\":null,\"errorCode\":null,…,\"eventName\":\"CreateDataset\",…}" },
  "attributes": [
    { "key": "trails.eventId", "value": { "stringValue": "0199a3f2-7c1d-7a4b-9e2f-1b3c4d5e6f70" } },
    { "key": "trails.eventName", "value": { "stringValue": "CreateDataset" } },
    { "key": "trails.eventSource", "value": { "stringValue": "obs.ezghcloud.com" } },
    { "key": "trails.eventType", "value": { "stringValue": "ApiCall" } },
    { "key": "trails.readOnly", "value": { "stringValue": "false" } },
    { "key": "trails.userIdentity.type", "value": { "stringValue": "User" } },
    { "key": "trails.userIdentity.principalId", "value": { "stringValue": "user_7f3k2m" } },
    { "key": "trails.userIdentity.userName", "value": { "stringValue": "ada@example.com" } },
    { "key": "trails.sourceIpAddress", "value": { "stringValue": "203.0.113.7" } },
    { "key": "trails.requestId", "value": { "stringValue": "req_1" } },
    { "key": "trails.organizationId", "value": { "stringValue": "01990000-0000-7000-8000-000000000001" } },
    { "key": "trails.resources", "value": { "stringValue": "ezgh:us-west-1:org_k3f9a0x2m7qp:prj_x:ds_web" } },
    { "key": "trails.deliveryId", "value": { "stringValue": "01990000-0000-7000-8000-0000000000d1" } }
  ]
}

Your project’s transform rules apply to delivered records like any others. Trails remains the record of what happened for its 30 days.

Search the logs

In the console, open Observability, then Logs, choose the delivery’s environment, and filter on the service ezgh.trails; a delivery’s View logs opens exactly that. Add attribute filters such as trails.eventName, trails.userIdentity.userName or trails.userIdentity.accessKeyId.

With the CLI:

# Every bot deleted in the last day.
ezgh obs search --project security --environment audit --from 1d \
  --service ezgh.trails --attr trails.eventName=DeleteBot

# Refused calls.
ezgh obs search --project security --environment audit --from 7d \
  --service ezgh.trails --attr trails.errorCode=access_denied

Status and dropped events

Status Means
active Delivering.
paused Turned off (enabled: false). Nothing is delivered until it’s resumed; events from while it was paused aren’t delivered.
failing The last attempt to deliver failed after the last success. lastError says why.

A delivery that can’t deliver (Observability unavailable, or the project’s limits refusing it) keeps its events and retries, with increasing waits, for up to 3 days. Events still undelivered after 3 days are dropped. droppedEvents counts every event that was never delivered: those dropped after 3 days, and records Observability rejected, such as one whose eventTime falls outside the dataset’s accepted window.

Delivery is at least once: after a retry, a record can arrive twice. Use trails.eventId to tell repeats apart.

Deleting the destination project, or the organization, deletes its deliveries. Deleting a delivery drops the events it hasn’t delivered; what it delivered stays in Observability.

Permissions

Action Allows In AdministratorAccess and TrailsFullAccess In ReadOnlyAccess
audit.deliveries.list List the organization’s deliveries Yes Yes
audit.deliveries.get View a delivery and its status Yes Yes
audit.deliveries.create Create a delivery Yes No
audit.deliveries.update Rename, pause or resume a delivery, or turn read events on or off Yes No
audit.deliveries.delete Delete a delivery Yes No

Creating a delivery, or turning enabled or includeReads on, sends events into the destination project’s Observability, so it also needs obs.logs.ingest on that project, as creating an ingest key does. Reading the delivered logs needs the usual Observability actions, such as obs.logs.search.

Creating, changing and deleting a delivery are recorded in Trails as CreateTrailDelivery, UpdateTrailDelivery and DeleteTrailDelivery, with the delivery (TrailDelivery) in resources. A delivery deleted with its project or organization is a service event by organizations.ezghcloud.com.

Navigation

Type to search…

↑↓ navigate↵ selectEsc close