---
title: "Span metrics"
description: "The request, error and latency series Observability generates from your traces, with their exact names, labels and buckets, and how to query them in PromQL and Grafana."
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.ezghcloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Span metrics

Observability generates metrics from the spans you send: a request count and a latency histogram
per service and operation (**span metrics**), and request, failure and latency series per edge of
the [service map](/observability/service-map) (**service-graph series**). They're ordinary
metrics in each environment's dataset, read with PromQL through the project's
[Prometheus API](/observability/metrics#query-with-promql) like the metrics you send. They power
the [service catalog](/observability/service-catalog)'s stats and the service map, and they make
long time ranges cheaper, because a chart reads the series instead of every span.

- **Automatic.** Every dataset that receives traces gets them, with nothing to configure.
- **Free.** They aren't counted in ingested or stored bytes, in a dataset's active series, or
  against ingest rate limits.
- **Kept like your metrics.** They follow the dataset's metrics retention class (13 months by
  default).
- **Tempo-compatible.** Series names, label names and buckets are the same as Grafana Tempo's
  metrics generator. Label values for services with a `service.namespace` are
  `namespace/name`, where Tempo uses `service.name` alone.

## Span metrics

Every span counts, of every kind, once per (service, span name, span kind, status):

| Series | Type | Value |
| --- | --- | --- |
| `traces_spanmetrics_calls_total` | Counter | Spans |
| `traces_spanmetrics_latency_bucket`, `traces_spanmetrics_latency_sum`, `traces_spanmetrics_latency_count` | Histogram, in seconds | Span durations |

| Label | Value |
| --- | --- |
| `service` | The span's [service key](/observability/service-catalog#how-services-are-named): `service.namespace/service.name`, or `service.name` |
| `span_name` | The span's name, or `(other)` past the [span-name cap](#caps) |
| `span_kind` | `SPAN_KIND_SERVER`, `SPAN_KIND_CLIENT`, `SPAN_KIND_PRODUCER`, `SPAN_KIND_CONSUMER`, `SPAN_KIND_INTERNAL` or `SPAN_KIND_UNSPECIFIED` |
| `status_code` | `STATUS_CODE_UNSET`, `STATUS_CODE_OK` or `STATUS_CODE_ERROR` |
| `job` | The service key, as for any OTLP metric |
| `environment` | The dataset's environment |
| `application` | The span's `ezgh.metadata.application` (`unknown` when it has none) |
| `le` | The bucket's upper bound, on `_bucket` series |

The latency buckets are Tempo's defaults, in seconds: `0.002`, `0.004`, `0.008`, `0.016`,
`0.032`, `0.064`, `0.128`, `0.256`, `0.512`, `1.024`, `2.048`, `4.096`, `8.192`, `16.384` and
`+Inf`. There's no `instance` label, so a service's pods share its series.

**Requests** into a service are its spans of kind server and consumer:
`span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER"`. Errors are those with
`status_code="STATUS_CODE_ERROR"`.

## Service-graph series

One request counts once per edge (client, server, connection type) of the
[service map](/observability/service-map):

| Series | Type | Value |
| --- | --- | --- |
| `traces_service_graph_request_total` | Counter | Requests |
| `traces_service_graph_request_failed_total` | Counter | Requests where either span has status `error` |
| `traces_service_graph_request_server_seconds_bucket`, `_sum`, `_count` | Histogram, in seconds | The server span's duration. Edges to services and from `user` |
| `traces_service_graph_request_client_seconds_bucket`, `_sum`, `_count` | Histogram, in seconds | The client span's duration. Edges between services and to virtual nodes |
| `traces_service_graph_unpaired_spans_total` | Counter | Client or server spans that weren't paired |
| `traces_service_graph_dropped_spans_total` | Counter | Spans not paired because too many were waiting |

| Label | Value |
| --- | --- |
| `client` | The caller's service key, or `user` for requests from outside |
| `server` | The callee's service key, or a [virtual node](/observability/service-map#virtual-nodes) such as `postgresql/orders` |
| `connection_type` | Absent between two services; `database`, `messaging_system` or `virtual_node` for virtual nodes and `user` |
| `client_application`, `server_application` | Each side's `ezgh.metadata.application`; absent for a virtual node's server and for `user` |
| `job` | `ezgh.servicegraph` |
| `environment` | The dataset's environment |
| `application` | `unknown` |
| `le` | The bucket's upper bound, on `_bucket` series |

The unpaired and dropped counters have only `client` or `server` (the service whose span it was)
and the platform labels. The latency buckets are Tempo's service-graph defaults, in seconds:
`0.1`, `0.2`, `0.4`, `0.8`, `1.6`, `3.2`, `6.4`, `12.8` and `+Inf`. A series appears once it
has something to count: `request_failed_total` only after an edge's first failure.

### As stored

The series are stored as cumulative OTLP metrics and translated to the names above when they're
read, like any [OTLP metric](/observability/metrics#names-and-labels):

| OTLP metric | Unit | Type |
| --- | --- | --- |
| `traces_spanmetrics_calls` | None | Sum, monotonic |
| `traces_spanmetrics_latency` | None (values in seconds) | Histogram |
| `traces_service_graph_request` | None | Sum, monotonic |
| `traces_service_graph_request_failed` | None | Sum, monotonic |
| `traces_service_graph_request_server_seconds` | `s` | Histogram |
| `traces_service_graph_request_client_seconds` | `s` | Histogram |
| `traces_service_graph_unpaired_spans` | None | Sum, monotonic |
| `traces_service_graph_dropped_spans` | None | Sum, monotonic |

Each series' resource is the service's `service.name` and `service.namespace` (for the
service-graph series, `ezgh.servicegraph`), with the environment and application, and nothing
else.

## Queries

These are the queries the console uses. Replace `production` with an environment, or use
`environment=~".+"` for all of them, and `5m` with the range you want.

```promql
# Requests per second into each service
sum by (service) (rate(traces_spanmetrics_calls_total{environment="production", span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER"}[5m]))

# Error ratio of each service's requests
sum by (service) (rate(traces_spanmetrics_calls_total{environment="production", span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER", status_code="STATUS_CODE_ERROR"}[5m]))
  / sum by (service) (rate(traces_spanmetrics_calls_total{environment="production", span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER"}[5m]))

# p95 latency of each service's requests
histogram_quantile(0.95, sum by (service, le) (rate(traces_spanmetrics_latency_bucket{environment="production", span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER"}[5m])))

# One service's slowest operations
topk(10, histogram_quantile(0.95, sum by (span_name, span_kind, le) (rate(traces_spanmetrics_latency_bucket{environment="production", service="payments/api"}[1h]))))

# What calls payments/api, and how often
sum by (client, connection_type) (rate(traces_service_graph_request_total{environment="production", server="payments/api"}[5m]))

# What payments/api calls, and its p95 as payments/api measures it
histogram_quantile(0.95, sum by (server, connection_type, le) (rate(traces_service_graph_request_client_seconds_bucket{environment="production", client="payments/api"}[5m])))
```

```sh
ezgh obs metrics query 'sum by (service) (rate(traces_spanmetrics_calls_total{environment="production", span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER"}[5m]))' --from 6h --step 5m --project web
```

- Read the counters with `rate` or `increase`, never as raw values: they restart from zero now
  and then (see [Accuracy](#accuracy)), which `rate` and `increase` handle.
- A point is written every 15 seconds for each series that changed. Use ranges of at least a
  minute, longer for services with little traffic: a series with no new spans in a range has no
  `rate` there.
- The [query budgets](/observability/metrics#query-budgets) apply. A busy series has a point every
  15 seconds, about 5,760 a day, and a latency histogram is 15 `_bucket` series per label set,
  so a query over weeks of many series can be over budget: use a larger step, more label
  matchers, or fewer series.

### Grafana

Use the project's Prometheus data source from [Metrics](/observability/metrics#grafana): the
series are there with nothing else to set up. In panels, use `$__rate_interval` as the range,
which is at least 1 minute with Grafana's default 15-second scrape interval, and `{{service}}`,
`{{span_name}}` or `{{client}} → {{server}}` as legends. For example, a time series panel of
request rates by service:

```promql
sum by (service) (rate(traces_spanmetrics_calls_total{environment="$environment", span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER"}[$__rate_interval]))
```

A variable of service keys is `label_values(traces_spanmetrics_calls_total, service)`. On
Observability's own dashboards, the **Service overview** template reads span metrics the same way.

## Caps

| Cap | Value | Past it |
| --- | --- | --- |
| Span names | 200 per service per environment, each hour | New names in that hour are counted as `span_name="(other)"` |
| Generated series | About 50,000 per environment | New span-metric series are counted with `span_name="(other)"`; new series that still don't fit, and new service-graph series, aren't written |

The span-name cap keeps span names that hold IDs or URLs, such as `GET /orders/8412`, from
creating a series each. Name spans after their route (`GET /orders/{id}`), as OpenTelemetry's
HTTP conventions do, and the cap is rarely reached. A series counts toward the series cap while
it has had spans in the last hour.

## Accuracy

Span metrics and the service graph are approximate by design, as Tempo's are:

- **They count the spans Observability receives.** If you sample traces before sending them (head
  sampling in an SDK, or a Collector's sampler), they count the spans you send: with 10% head
  sampling, about a tenth of the requests. [Trace sampling](#sampling-doesnt-change-these-counts)
  in Observability doesn't reduce them.
- **They start when the spans arrive.** Spans stored before span metrics were available aren't
  counted, and the series follow new spans by about a minute.
- **An interruption on Observability's side** can count up to 15 seconds of spans twice, and
  restarts the counters from zero, which `rate` and `increase` treat as a reset. A series with no
  spans for an hour also starts again from zero.
- **Spans stored straight to cold storage generate nothing:** spans sent with
  `x-ezgh-placement: cold`, to a dataset whose placement is cold (such as `cold-365d`), or whose
  time is already past the dataset's hot days when they arrive. Rarely, spans that arrive while
  Observability is moving a dataset's storage generate nothing either.
- **Spans whose service key isn't cataloged** (one starting with `ezgh.`, longer than 255 bytes,
  or with control characters) generate nothing.
- **Pairing** misses edges whose spans arrive more than 10 seconds apart or cross environments;
  see [What isn't paired](/observability/service-map#what-isnt-paired).

## Sampling doesn't change these counts

In a dataset with [trace sampling](/observability/sampling), span metrics and the service-graph
series count every span Observability receives, whether its trace is kept or dropped, and don't
count kept spans a second time.

- They follow the spans by about a decision wait longer than usual (30 seconds by default, up to
  about 2 minutes), because a sampled dataset's spans are counted after the decision wait.
- Searches, trace reads and dashboards over raw traces count kept spans only. A service's
  request rate from span metrics is therefore higher than the requests in its traces; the
  console marks this with a **Sampled** badge.

## Access and Trails

Reading the series is a PromQL query and needs `obs.metrics.query` on the dataset, which
`ObsFullAccess`, `ObsReadOnlyAccess` and `ReadOnlyAccess` include. Queries aren't recorded in
[Trails](/trails).

Source: https://docs.ezghcloud.com/observability/span-metrics/index.mdx