Traces

Observability stores the traces your services send over OpenTelemetry (OTLP), next to their logs. Spans go to the dataset for their environment, are searchable within seconds, and stay readable for the dataset’s retention class.

Send traces

Any OTLP trace exporter works: the OpenTelemetry Collector, SDKs and language agents. Set the base endpoint and the ingest key header; the exporter adds the traces path itself.

OTEL_EXPORTER_OTLP_ENDPOINT=https://obs-ingest.ezghcloud.com
OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer%20ezgh_ik_…"
OTEL_TRACES_EXPORTER=otlp
OTEL_RESOURCE_ATTRIBUTES="service.name=checkout,ezgh.metadata.application=storefront,ezgh.metadata.environment=production"

This works with OTEL_EXPORTER_OTLP_PROTOCOL set to grpc, http/protobuf or http/json. OTEL_EXPORTER_OTLP_HEADERS uses the W3C Baggage format, so the space after Bearer is written %20.

The ingest key must allow obs.traces.ingest. A key that doesn’t is refused with 403; create a key that allows both obs.logs.ingest and obs.traces.ingest, or a second key.

With the Collector

The example uses the current exporter name, otlp_grpc. Collectors released before the rename call it otlp.

processors:
  resource:
    attributes:
      - { key: ezgh.metadata.environment, value: production, action: insert }
exporters:
  otlp_grpc/ezgh:
    endpoint: obs-ingest.ezghcloud.com:443
    headers: { authorization: "Bearer ${env:EZGH_INGEST_KEY}" }
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resource]
      exporters: [otlp_grpc/ezgh]

See Send from a Collector for batching, retries and routing to several projects.

Environments and datasets

Each span goes to the dataset for its resource’s ezgh.metadata.environment, created automatically the first time an environment sends data. A span without the attribute goes to the unknown dataset. See Service metadata.

Limits

A span that breaks a limit is rejected; the rest of the request is stored, and the response’s partial_success says how many spans were rejected and why.

A trace ID must be 16 bytes and a span ID 8 bytes, neither all zero.

Search spans

Search a dataset’s spans whose start time is between from and to. Filters, all optional, narrow the search: services, applications, spanNames, kinds, statuses, traceId, minDurationNanos, maxDurationNanos, rootOnly and attributes. Results are spans, newest first, a page at a time. rootOnly returns only root spans, the ones without a parent.

{
  "from": "2026-10-02T10:00:00Z",
  "to": "2026-10-02T12:00:00Z",
  "filters": {
    "services": ["checkout"],
    "statuses": ["error"],
    "minDurationNanos": 250000000,
    "attributes": [{ "key": "http.route", "op": "equals", "value": "/cart" }]
  },
  "limit": 100
}

• An attribute filter’s op is equals (with a value) or exists. Its optional scope is record, the span’s own attributes (the default), or resource, its resource attributes.
• limit is 1 to 100, 50 by default.
• To read the next page, send only cursor (the previous page’s nextCursor) and limit.
• A search has a budget of time and data read. A search over it is refused with 422 query_too_expensive; narrow the time range or add filters.

Read a trace

Get a trace returns the spans of one trace ID in a dataset, oldest first, with their events, links and scope.

• Without from and to, it reads spans that started in the last 7 days. The range can span at most 31 days.
• Spans come a page at a time: limit is 1 to 10,000, 1,000 by default. Pass the previous page’s nextCursor as cursor to read the next page.
• A trace with no spans in the range, or an unknown trace ID, returns 200 with no spans. Check visibleThrough: spans sent in the last few seconds may not be searchable yet.

An exporter that retries after a lost response can store a span twice; both copies are returned, with the same spanId.

Sampled datasets

A dataset with trace sampling on stores only the traces its rules keep, deciding each trace whole after its spans arrive. Sending is unchanged: every span is accepted as usual, and the response doesn’t say whether its trace was kept.

• Searches and trace reads return kept spans only, so counts, error rates and latencies from them are of stored traces. A dropped trace’s ID usually returns no spans.
• A kept trace is searchable about the dataset’s decision wait (30 seconds by default) after its first span arrives.
• The console’s Traces page shows a Sampled badge: “Counts on this page are of stored traces”.
• Span metrics and the service map count every span, kept or dropped, so a service’s request rate there is higher than the requests in its traces.

Access

ObsFullAccess includes all three. ObsReadOnlyAccess and ReadOnlyAccess include obs.traces.search and obs.traces.getSpans, so read-only users can search spans and read traces. Searches and trace reads aren’t recorded in Trails.

Pricing

Traces are metered by ingested bytes and by stored bytes, hot and cold. In a sampled dataset, only kept spans are metered. These meters aren’t billed yet.