---
title: "Traces"
description: "Send OpenTelemetry traces with a project ingest key, then search spans and read whole traces."
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.ezghcloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Traces

Observability stores the traces your services send over OpenTelemetry (OTLP), next to their logs.
Spans go to the dataset for their environment, are searchable within seconds, and stay readable
for the dataset's retention class.

## Send traces

Any OTLP trace exporter works: the OpenTelemetry Collector, SDKs and language agents. Set the
base endpoint and the ingest key header; the exporter adds the traces path itself.

```sh
OTEL_EXPORTER_OTLP_ENDPOINT=https://obs-ingest.ezghcloud.com
OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer%20ezgh_ik_…"
OTEL_TRACES_EXPORTER=otlp
OTEL_RESOURCE_ATTRIBUTES="service.name=checkout,ezgh.metadata.application=storefront,ezgh.metadata.environment=production"
```

This works with `OTEL_EXPORTER_OTLP_PROTOCOL` set to `grpc`, `http/protobuf` or `http/json`.
`OTEL_EXPORTER_OTLP_HEADERS` uses the W3C Baggage format, so the space after `Bearer` is written
`%20`.

| Protocol | Endpoint |
| --- | --- |
| OTLP/gRPC | `obs-ingest.ezghcloud.com:443`, method `opentelemetry.proto.collector.trace.v1.TraceService/Export` |
| OTLP/HTTP | `POST https://obs-ingest.ezghcloud.com/v1/traces`, protobuf or JSON, optionally gzip |

The ingest key must allow `obs.traces.ingest`. A key that doesn't is refused with `403`; create a
key that allows both `obs.logs.ingest` and `obs.traces.ingest`, or a second key.

### With the Collector

The example uses the current exporter name, `otlp_grpc`. Collectors released before the rename
call it `otlp`.

```yaml
processors:
  resource:
    attributes:
      - { key: ezgh.metadata.environment, value: production, action: insert }
exporters:
  otlp_grpc/ezgh:
    endpoint: obs-ingest.ezghcloud.com:443
    headers: { authorization: "Bearer ${env:EZGH_INGEST_KEY}" }
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resource]
      exporters: [otlp_grpc/ezgh]
```

See [Send from a Collector](/observability/send-with-collector) for batching, retries and
routing to several projects.

## Environments and datasets

Each span goes to the dataset for its resource's `ezgh.metadata.environment`, created
automatically the first time an environment sends data. A span without the attribute goes to the
`unknown` dataset. See [Service metadata](/observability#service-metadata).

## Limits

A span that breaks a limit is rejected; the rest of the request is stored, and the response's
`partial_success` says how many spans were rejected and why.

| Limit | Value |
| --- | --- |
| Request body | 5 MiB compressed, 64 MiB uncompressed |
| Span size | 64 KiB, with its resource and scope |
| Span name | 4 KiB |
| Events per span | 128 |
| Links per span | 128 |
| Attributes per map | 256 (span, resource, each event and each link) |
| Span duration | about 104 days |
| Start time | within the dataset's retention, and at most 1 day ahead |

A trace ID must be 16 bytes and a span ID 8 bytes, neither all zero.

## Search spans

[Search a dataset's spans](/obs-api/SearchTraces/) whose start time is between `from` and
`to`. Filters, all optional, narrow the search: `services`, `applications`, `spanNames`, `kinds`,
`statuses`, `traceId`, `minDurationNanos`, `maxDurationNanos`, `rootOnly` and `attributes`.
Results are spans, newest first, a page at a time. `rootOnly` returns only root spans, the ones
without a parent.

```json
{
  "from": "2026-10-02T10:00:00Z",
  "to": "2026-10-02T12:00:00Z",
  "filters": {
    "services": ["checkout"],
    "statuses": ["error"],
    "minDurationNanos": 250000000,
    "attributes": [{ "key": "http.route", "op": "equals", "value": "/cart" }]
  },
  "limit": 100
}
```

- An attribute filter's `op` is `equals` (with a `value`) or `exists`. Its optional `scope` is
  `record`, the span's own attributes (the default), or `resource`, its resource attributes.
- `limit` is 1 to 100, 50 by default.
- To read the next page, send only `cursor` (the previous page's `nextCursor`) and `limit`.
- A search has a budget of time and data read. A search over it is refused with
  `422 query_too_expensive`; narrow the time range or add filters.

## Read a trace

[Get a trace](/obs-api/GetTrace/) returns the spans of one trace ID in a dataset, oldest
first, with their events, links and scope.

- Without `from` and `to`, it reads spans that started in the last 7 days. The range can span at
  most 31 days.
- Spans come a page at a time: `limit` is 1 to 10,000, 1,000 by default. Pass the previous page's
  `nextCursor` as `cursor` to read the next page.
- A trace with no spans in the range, or an unknown trace ID, returns `200` with no spans. Check
  `visibleThrough`: spans sent in the last few seconds may not be searchable yet.

An exporter that retries after a lost response can store a span twice; both copies are returned,
with the same `spanId`.

## Sampled datasets

A dataset with [trace sampling](/observability/sampling) on stores only the traces its rules keep,
deciding each trace whole after its spans arrive. Sending is unchanged: every span is accepted as
usual, and the response doesn't say whether its trace was kept.

- Searches and trace reads return kept spans only, so counts, error rates and latencies from them
  are of stored traces. A dropped trace's ID usually returns no spans.
- A kept trace is searchable about the dataset's decision wait (30 seconds by default) after its
  first span arrives.
- The console's Traces page shows a **Sampled** badge: "Counts on this page are of stored traces".
- [Span metrics](/observability/span-metrics) and the [service map](/observability/service-map)
  count every span, kept or dropped, so a service's request rate there is higher than the
  requests in its traces.

## Access

| Action | Allows |
| --- | --- |
| `obs.traces.ingest` | Sending traces, with a project ingest key |
| `obs.traces.search` | Searching spans (returns span content) |
| `obs.traces.getSpans` | Reading whole traces (returns span content) |

`ObsFullAccess` includes all three. `ObsReadOnlyAccess` and `ReadOnlyAccess` include
`obs.traces.search` and `obs.traces.getSpans`, so read-only users can search spans and read
traces. Searches and trace reads aren't recorded in [Trails](/trails).

## Pricing

Traces are metered by ingested bytes and by stored bytes, hot and cold. In a
[sampled dataset](/observability/sampling#billing-and-limits), only kept spans are metered. These
meters aren't billed yet.

Source: https://docs.ezghcloud.com/observability/traces/index.mdx