Span metrics

Observability generates metrics from the spans you send: a request count and a latency histogram per service and operation (span metrics), and request, failure and latency series per edge of the service map (service-graph series). They’re ordinary metrics in each environment’s dataset, read with PromQL through the project’s Prometheus API like the metrics you send. They power the service catalog’s stats and the service map, and they make long time ranges cheaper, because a chart reads the series instead of every span.

• Automatic. Every dataset that receives traces gets them, with nothing to configure.
• Free. They aren’t counted in ingested or stored bytes, in a dataset’s active series, or against ingest rate limits.
• Kept like your metrics. They follow the dataset’s metrics retention class (13 months by default).
• Tempo-compatible. Series names, label names and buckets are the same as Grafana Tempo’s metrics generator. Label values for services with a service.namespace are namespace/name, where Tempo uses service.name alone.

Span metrics

Every span counts, of every kind, once per (service, span name, span kind, status):

The latency buckets are Tempo’s defaults, in seconds: 0.002, 0.004, 0.008, 0.016, 0.032, 0.064, 0.128, 0.256, 0.512, 1.024, 2.048, 4.096, 8.192, 16.384 and +Inf. There’s no instance label, so a service’s pods share its series.

Requests into a service are its spans of kind server and consumer: span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER". Errors are those with status_code="STATUS_CODE_ERROR".

Service-graph series

One request counts once per edge (client, server, connection type) of the service map:

The unpaired and dropped counters have only client or server (the service whose span it was) and the platform labels. The latency buckets are Tempo’s service-graph defaults, in seconds: 0.1, 0.2, 0.4, 0.8, 1.6, 3.2, 6.4, 12.8 and +Inf. A series appears once it has something to count: request_failed_total only after an edge’s first failure.

As stored

The series are stored as cumulative OTLP metrics and translated to the names above when they’re read, like any OTLP metric:

Each series’ resource is the service’s service.name and service.namespace (for the service-graph series, ezgh.servicegraph), with the environment and application, and nothing else.

Queries

These are the queries the console uses. Replace production with an environment, or use environment=~".+" for all of them, and 5m with the range you want.

# Requests per second into each service
sum by (service) (rate(traces_spanmetrics_calls_total{environment="production", span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER"}[5m]))

# Error ratio of each service's requests
sum by (service) (rate(traces_spanmetrics_calls_total{environment="production", span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER", status_code="STATUS_CODE_ERROR"}[5m]))
  / sum by (service) (rate(traces_spanmetrics_calls_total{environment="production", span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER"}[5m]))

# p95 latency of each service's requests
histogram_quantile(0.95, sum by (service, le) (rate(traces_spanmetrics_latency_bucket{environment="production", span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER"}[5m])))

# One service's slowest operations
topk(10, histogram_quantile(0.95, sum by (span_name, span_kind, le) (rate(traces_spanmetrics_latency_bucket{environment="production", service="payments/api"}[1h]))))

# What calls payments/api, and how often
sum by (client, connection_type) (rate(traces_service_graph_request_total{environment="production", server="payments/api"}[5m]))

# What payments/api calls, and its p95 as payments/api measures it
histogram_quantile(0.95, sum by (server, connection_type, le) (rate(traces_service_graph_request_client_seconds_bucket{environment="production", client="payments/api"}[5m])))

ezgh obs metrics query 'sum by (service) (rate(traces_spanmetrics_calls_total{environment="production", span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER"}[5m]))' --from 6h --step 5m --project web

• Read the counters with rate or increase, never as raw values: they restart from zero now and then (see Accuracy), which rate and increase handle.
• A point is written every 15 seconds for each series that changed. Use ranges of at least a minute, longer for services with little traffic: a series with no new spans in a range has no rate there.
• The query budgets apply. A busy series has a point every 15 seconds, about 5,760 a day, and a latency histogram is 15 _bucket series per label set, so a query over weeks of many series can be over budget: use a larger step, more label matchers, or fewer series.

Grafana

Use the project’s Prometheus data source from Metrics: the series are there with nothing else to set up. In panels, use $__rate_interval as the range, which is at least 1 minute with Grafana’s default 15-second scrape interval, and {{service}}, {{span_name}} or {{client}} → {{server}} as legends. For example, a time series panel of request rates by service:

sum by (service) (rate(traces_spanmetrics_calls_total{environment="$environment", span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER"}[$__rate_interval]))

A variable of service keys is label_values(traces_spanmetrics_calls_total, service). On Observability’s own dashboards, the Service overview template reads span metrics the same way.

Caps

The span-name cap keeps span names that hold IDs or URLs, such as GET /orders/8412, from creating a series each. Name spans after their route (GET /orders/{id}), as OpenTelemetry’s HTTP conventions do, and the cap is rarely reached. A series counts toward the series cap while it has had spans in the last hour.

Accuracy

Span metrics and the service graph are approximate by design, as Tempo’s are:

• They count the spans Observability receives. If you sample traces before sending them (head sampling in an SDK, or a Collector’s sampler), they count the spans you send: with 10% head sampling, about a tenth of the requests. Trace sampling in Observability doesn’t reduce them.
• They start when the spans arrive. Spans stored before span metrics were available aren’t counted, and the series follow new spans by about a minute.
• An interruption on Observability’s side can count up to 15 seconds of spans twice, and restarts the counters from zero, which rate and increase treat as a reset. A series with no spans for an hour also starts again from zero.
• Spans stored straight to cold storage generate nothing: spans sent with x-ezgh-placement: cold, to a dataset whose placement is cold (such as cold-365d), or whose time is already past the dataset’s hot days when they arrive. Rarely, spans that arrive while Observability is moving a dataset’s storage generate nothing either.
• Spans whose service key isn’t cataloged (one starting with ezgh., longer than 255 bytes, or with control characters) generate nothing.
• Pairing misses edges whose spans arrive more than 10 seconds apart or cross environments; see What isn’t paired.

Sampling doesn’t change these counts

In a dataset with trace sampling, span metrics and the service-graph series count every span Observability receives, whether its trace is kept or dropped, and don’t count kept spans a second time.

• They follow the spans by about a decision wait longer than usual (30 seconds by default, up to about 2 minutes), because a sampled dataset’s spans are counted after the decision wait.
• Searches, trace reads and dashboards over raw traces count kept spans only. A service’s request rate from span metrics is therefore higher than the requests in its traces; the console marks this with a Sampled badge.

Access and Trails

Reading the series is a PromQL query and needs obs.metrics.query on the dataset, which ObsFullAccess, ObsReadOnlyAccess and ReadOnlyAccess include. Queries aren’t recorded in Trails.