---
title: "Metrics"
description: "Send OpenTelemetry metrics or Prometheus remote write with a project ingest key, then query them with PromQL from Grafana or the CLI."
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.ezghcloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Metrics

Observability stores the metrics your services send over OpenTelemetry (OTLP) or Prometheus
remote write, next to their logs and traces. Metrics go to the dataset for their environment and
are read with PromQL through a Prometheus-compatible API, so Grafana's Prometheus data source and
other Prometheus clients work unchanged.

## Send OTLP metrics

Any OTLP metric exporter works: the OpenTelemetry Collector, SDKs and language agents. Set the
base endpoint and the ingest key header; the exporter adds the metrics path itself.

```sh
OTEL_EXPORTER_OTLP_ENDPOINT=https://obs-ingest.ezghcloud.com
OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer%20ezgh_ik_…"
OTEL_METRICS_EXPORTER=otlp
OTEL_RESOURCE_ATTRIBUTES="service.name=checkout,ezgh.metadata.application=storefront,ezgh.metadata.environment=production"
```

This works with `OTEL_EXPORTER_OTLP_PROTOCOL` set to `grpc`, `http/protobuf` or `http/json`.
`OTEL_EXPORTER_OTLP_HEADERS` uses the W3C Baggage format, so the space after `Bearer` is written
`%20`.

| Protocol | Endpoint |
| --- | --- |
| OTLP/gRPC | `obs-ingest.ezghcloud.com:443`, method `opentelemetry.proto.collector.metrics.v1.MetricsService/Export` |
| OTLP/HTTP | `POST https://obs-ingest.ezghcloud.com/v1/metrics`, protobuf or JSON, optionally gzip |

The ingest key must allow `obs.metrics.ingest`. A key that doesn't is refused with `403`; create
a key that allows it, or a second key:

```sh
ezgh ingest-keys create otel-collector --action obs.logs.ingest,obs.traces.ingest,obs.metrics.ingest --project web
```

- All five OTLP point types are stored: gauges, sums, histograms, exponential histograms and
  summaries, with their exemplars.
- Cumulative and delta temporality are both accepted and stored as sent. A sum or histogram with
  unspecified temporality is rejected.
- A point with the `no recorded value` flag is a Prometheus staleness marker: the series ends
  there.

### With the Collector

The example uses the current exporter name, `otlp_grpc`. Collectors released before the rename
call it `otlp`. A Collector's `prometheus` receiver can scrape Prometheus targets into the same
pipeline.

```yaml
processors:
  resource:
    attributes:
      - { key: ezgh.metadata.environment, value: production, action: insert }
exporters:
  otlp_grpc/ezgh:
    endpoint: obs-ingest.ezghcloud.com:443
    headers: { authorization: "Bearer ${env:EZGH_INGEST_KEY}" }
service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [resource]
      exporters: [otlp_grpc/ezgh]
```

See [Send from a Collector](/observability/send-with-collector) for batching, retries and
routing to several projects.

## Send with Prometheus remote write

A Prometheus server (or any remote-write sender) writes to
`POST https://obs-ingest.ezghcloud.com/api/v1/write`, with the ingest key as a bearer token.
Remote write 1.0 and 2.0 are both accepted, chosen by the request's `Content-Type`.

```yaml
global:
  external_labels:
    ezgh_environment: production
    ezgh_application: storefront
remote_write:
  - url: https://obs-ingest.ezghcloud.com/api/v1/write
    protobuf_message: io.prometheus.write.v2.Request
    authorization:
      credentials_file: /etc/prometheus/ezgh-ingest-key
```

- **Environment and application** come from the `ezgh_environment` and `ezgh_application`
  labels, normalized like the `ezgh.metadata.*` resource attributes (a missing or invalid value
  is `unknown`). They're removed from the stored labels and read back as `environment` and
  `application`. Set them with `external_labels`, as above, or per target with relabeling.
- **Use remote write 2.0** (`protobuf_message: io.prometheus.write.v2.Request`, Prometheus 3.0
  and later) where you can: it carries each series' type, unit and help. Remote write 1.0
  samples are stored with the type `unknown`.
- Only bearer authorization is accepted; `basic_auth` and `sigv4` are refused with `401`.
- Classic histograms and summaries arrive as their `_bucket`, `_sum`, `_count` and
  `{quantile}` series and are stored that way. Native histograms (schema −4 to 8, and custom
  buckets) are read back as native histograms.

| Answer | Meaning | The sender |
| --- | --- | --- |
| `204` | Every sample and histogram was stored | Continues |
| `400` | Some or all samples were refused: the body says how many and why. The rest were stored | Drops the batch, without retrying |
| `401`, `403` | The key is missing, invalid or revoked, or doesn't allow `obs.metrics.ingest` | Drops the batch |
| `413` | Over 5 MiB compressed, or 64 MiB decompressed | Drops the batch |
| `415` | A `Content-Type` or `Content-Encoding` other than remote write's protobuf and `snappy` | Drops the batch |
| `429`, `503` | Over a rate, or briefly unavailable, with `Retry-After` | Retries |

Remote write 2.0 answers also carry `X-Prometheus-Remote-Write-Samples-Written`,
`…-Histograms-Written` and `…-Exemplars-Written`.

## Environments and datasets

Each point goes to the dataset for its environment, created automatically the first time an
environment sends data, as for logs and traces. A point without an environment goes to the
`unknown` dataset. See [Service metadata](/observability#service-metadata).

## Query with PromQL

Each project has a Prometheus HTTP API at

```text
https://obs.ezghcloud.com/v1/organizations/<organization ID>/projects/<project ID>/obs/prometheus
```

`ezgh projects get <project> -o json` shows both IDs. A query reads every dataset (environment)
of the project that the caller may query, so one query can compare environments; select one
with an `environment` matcher, such as `up{environment="production"}`.

| Path, under the base URL | Methods | Operation |
| --- | --- | --- |
| `/api/v1/query` | `GET`, `POST` | Instant query: [QueryMetrics](/obs-api/QueryMetrics/), [QueryMetricsPost](/obs-api/QueryMetricsPost/) |
| `/api/v1/query_range` | `GET`, `POST` | Range query: [QueryMetricsRange](/obs-api/QueryMetricsRange/), [QueryMetricsRangePost](/obs-api/QueryMetricsRangePost/) |
| `/api/v1/series` | `GET`, `POST` | Series matching selectors: [ListMetricSeries](/obs-api/ListMetricSeries/), [ListMetricSeriesPost](/obs-api/ListMetricSeriesPost/) |
| `/api/v1/labels` | `GET`, `POST` | Label names: [ListMetricLabels](/obs-api/ListMetricLabels/), [ListMetricLabelsPost](/obs-api/ListMetricLabelsPost/) |
| `/api/v1/label/<name>/values` | `GET` | One label's values: [ListMetricLabelValues](/obs-api/ListMetricLabelValues/) |
| `/api/v1/metadata` | `GET` | Each metric's type, unit and help: [GetMetricMetadata](/obs-api/GetMetricMetadata/) |
| `/api/v1/query_exemplars` | `GET`, `POST` | Exemplars, with trace and span IDs: [QueryMetricExemplars](/obs-api/QueryMetricExemplars/), [QueryMetricExemplarsPost](/obs-api/QueryMetricExemplarsPost/) |
| `/api/v1/status/buildinfo` | `GET` | The Prometheus version the API is compatible with: [GetMetricsBuildInfo](/obs-api/GetMetricsBuildInfo/) |

- Requests and answers are Prometheus's: `POST` bodies are form-encoded, times are RFC 3339 or
  Unix seconds, and answers are `{"status": "success", "data": …}`. Errors are
  `{"status": "error", "errorType": …, "error": …}` with an added `code`, the platform's error
  code.
- Authenticate with `Authorization: Bearer` and an [API key](/iam/api-keys), a
  [bot's API key](/iam/bots#create-an-api-key-for-a-bot), or an OAuth access token. Ingest keys
  are refused.
- Without `start`, `labels`, `label/<name>/values`, `series`, `metadata` and `query_exemplars`
  cover the last 6 hours, not all time.
- The lookback delta is 5 minutes, as in Prometheus. A query's `lookback_delta` can set 1 second
  to 1 hour.

### Grafana

Add a **Prometheus** data source with the project's base URL and a bot's API key:

1. Create a bot for Grafana and a policy that allows only `obs.metrics.query` in the project,
   then a key for the bot:

   ```sh
   ezgh iam bots create grafana
   ezgh iam policies create metrics-query --document @metrics-query.json
   ezgh iam policies attach metrics-query --bot grafana
   ezgh iam api-keys create grafana --bot grafana --expires-in-days 365
   ```

   ```json title="metrics-query.json"
   {
     "statements": [
       {
         "effect": "allow",
         "actions": ["obs.metrics.query"],
         "resources": [
           "ezgh:*:org_k3f9a0x2m7qp:prj_g7h8i9j0k1l2",
           "ezgh:*:org_k3f9a0x2m7qp:prj_g7h8i9j0k1l2:*"
         ]
       }
     ]
   }
   ```

2. In Grafana, open **Connections** > **Data sources** > **Add data source** and choose
   **Prometheus**.
3. Set **Prometheus server URL** to the project's base URL.
4. Under **HTTP headers**, add the header `Authorization` with the value `Bearer ezgh_…` (the
   bot's key). Leave **Authentication** as **No Authentication**.
5. Under **Advanced settings**, keep **HTTP method** `POST`, and set **Prometheus type** to
   **Prometheus**.
6. Select **Save & test**.

Or provision it:

```yaml
apiVersion: 1
datasources:
  - name: EZGH web
    type: prometheus
    access: proxy
    url: https://obs.ezghcloud.com/v1/organizations/<organization ID>/projects/<project ID>/obs/prometheus
    jsonData:
      httpMethod: POST
      prometheusType: Prometheus
      httpHeaderName1: Authorization
    secureJsonData:
      httpHeaderValue1: Bearer ${EZGH_GRAFANA_KEY}
```

One data source reads one project. Grafana alerting works against it like any Prometheus.

### From the CLI

```sh
ezgh obs metrics query 'sum by (job) (rate(http_server_request_duration_seconds_count[5m]))' --project web
ezgh obs metrics query 'up{environment="production"}' --from 6h --step 5m --project web
ezgh obs metrics label-values __name__ --environment production --project web
```

See [`ezgh obs metrics`](/cli/reference/obs-metrics). `-o json` prints the API's answer.

## Names and labels

OTLP metric names and attributes are translated to Prometheus names when they're read, the way
Prometheus's own OTLP receiver translates them by default. What was sent is stored unchanged.

| OTLP metric | Unit | Type | PromQL names |
| --- | --- | --- | --- |
| `http.server.request.duration` | `s` | Histogram | `http_server_request_duration_seconds_bucket` (with `le`), `…_sum`, `…_count` |
| `http.server.request.duration` | `s` | Exponential histogram | `http_server_request_duration_seconds`, a native histogram |
| `process.cpu.time` | `s` | Sum, monotonic | `process_cpu_time_seconds_total` |
| `system.memory.usage` | `By` | Sum, not monotonic | `system_memory_usage_bytes`, a gauge |
| `http.server.active_requests` | `{request}` | Sum, not monotonic | `http_server_active_requests`, a gauge |
| `rpc.server.duration` | `ms` | Summary | `rpc_server_duration_milliseconds` (with `quantile`), `…_sum`, `…_count` |

- Characters other than letters, digits and `_` become `_`; the unit becomes a suffix
  (annotations in braces are dropped), and a monotonic sum gets `_total`.
- **Labels** are the point's attributes, translated the same way (`http.response.status_code`
  is `http_response_status_code`). Two attributes that translate to the same label have their
  values joined with `;`.
- **Every series also has** `environment` and `application` (from `ezgh.metadata.*`), `job`
  (`service.namespace/service.name`, or `service.name`) and `instance` (`service.instance.id`).
  A sent attribute or remote-write label named `environment` or `application` with a different
  value is read as `exported_environment` or `exported_application`, so `environment` always
  names the dataset.
- **Other resource attributes** are in `target_info`: one series per resource, value `1`, with
  the resource attributes as labels plus `environment`, `application`, `job` and `instance`.
  Join it to add them:

  ```promql
  sum by (k8s_namespace_name) (
    rate(http_server_request_duration_seconds_count[5m])
    * on (environment, job, instance) group_left (k8s_namespace_name) target_info
  )
  ```

- The instrumentation scope's name and version aren't labels. Series that differ only in scope
  or in a resource attribute other than `job` and `instance` read as one series; the answer
  then carries a warning that series were merged. [EZGH Obs Agent](/observability/kubernetes-agent#kubernetes-labels-on-metrics)
  0.3.0 copies the Kubernetes resource attributes onto each point, so its series have
  `k8s_cluster_name`, `k8s_node_name`, `k8s_pod_name` and the other Kubernetes labels.
- **Delta** sums and histograms are accumulated from the start of the query, so `rate`,
  `increase` and `histogram_quantile` are exact. A delta counter's raw value counts from the
  query's start.
- **Remote-write** series keep their names and labels as sent, apart from `ezgh_environment`
  and `ezgh_application`.
- **Generated series.** Every dataset that receives traces also has the `traces_spanmetrics_*`
  and `traces_service_graph_*` series that Observability derives from its spans. They're free;
  see [Span metrics](/observability/span-metrics).

## Limits

A point that breaks a limit is rejected; the rest of the request is stored. OTLP answers say how
many points were rejected and why in `partial_success`; remote write answers `400`.

| Limit | Value |
| --- | --- |
| Request body | 5 MiB compressed, 64 MiB uncompressed |
| Point size | 64 KiB, with its resource and scope |
| Metric name | 1 KiB |
| Unit | 256 bytes |
| Attributes per map | 256 (point, resource, scope and each exemplar) |
| Explicit histogram buckets | 160 bounds |
| Exponential histogram | Scale −4 to 20 (above 8 is read at scale 8), 160 buckets per range |
| Summary quantiles | 64, each between 0 and 1 |
| Exemplars per point | 16 |
| Point time | Within the dataset's metrics retention, and at most 1 day ahead |

Counts at or above 2<sup>63</sup> are rejected. From remote write, gauge native histograms and
float histograms with fractional counts are rejected.

### Query budgets

Every query reads stored points, so a long range at a short step can be expensive. A query over a
budget is refused with `422` and `code` `query_too_expensive`, with `budget` naming which one,
never cut short. Narrow it: a shorter range, a larger step, or more label matchers.

| Budget | Value | `budget` |
| --- | --- | --- |
| Samples in memory while evaluating | 5,000,000 | `samples` |
| Evaluation time | 30 seconds (a query's `timeout` can only lower it) | `time` |
| Points read | 20,000,000 per request | `fetchedSamples` |
| Series | 100,000 per request | `series` |
| Data read | 20 GiB per request | `bytes` |
| Sample data fetched | 128 MiB per request | `memory` |
| Answer size | 64 MiB | `response` |
| Metric names, label values, exemplars | 100,000, 50,000, 10,000 | `rows` |

| Other limit | Value |
| --- | --- |
| Points per range query | 11,000 (`(end − start) / step`); more is `400` |
| `match[]` selectors | 1 to 20 |
| Query text | 64 KiB; a `POST` body 256 KiB (`413`) |
| Queries running at once | 20 per organization. A query waits up to 2 seconds for a turn, then is refused with `429` and `Retry-After`. A query can also get `429` when the service is busy |

| Status | `errorType` | `code` | When |
| --- | --- | --- | --- |
| `400` | `bad_data` | `invalid_request` | A parameter or the PromQL is invalid, or too many points |
| `401` | `unauthorized` | `unauthenticated` | No valid credential |
| `403` | `forbidden` | `access_denied` | The caller may query none of the project's datasets, or none of those the query's `environment` matchers select |
| `404` | `not_found` | `not_found` | The organization or project doesn't exist, or the caller can't see it |
| `422` | `execution` or `timeout` | `query_too_expensive` | Over a budget |
| `422` | `execution` | `query_failed` | The query failed to evaluate, as Prometheus answers it |
| `429` | `unavailable` | `too_many_requests` | Too many queries at once, with `Retry-After` |
| `503` | `unavailable` | `unavailable` | Briefly unavailable, with `Retry-After` |

## Retention

Each signal of a dataset has its own retention class. Metrics default to `14d-395d`, 13 months,
so a year can be compared with the last; logs and traces default to `7d-30d`. Every class can be
used for every signal.

| Class | Hot days | Total days |
| --- | --- | --- |
| `7d-30d` | 7 | 30 |
| `7d-90d` | 7 | 90 |
| `14d-365d` | 14 | 365 |
| `14d-395d` | 14 | 395 |
| `cold-365d` | 0 | 365 |

- A dataset's `retention` holds each signal's class: `{"logs": "7d-30d", "traces": "7d-30d",
  "metrics": "14d-395d"}`. `retentionClass` is the logs class, which traces follow unless set
  separately; changing it never changes the metrics class.
- Set one signal's class with `retention` in [CreateDataset](/obs-api/CreateDataset/) or
  [UpdateDataset](/obs-api/UpdateDataset/), such as `{"retention": {"metrics": "14d-365d"}}`.
  Where this isn't available yet, the request is refused with `409 signal_retention_unavailable`.
- A new class applies to points that arrive afterward; stored points keep the class they
  arrived with.
- Points are kept at full resolution for the whole class: there's no downsampling.
- `ezgh obs datasets get` shows every signal's class, and
  [ListRetentionClasses](/obs-api/ListRetentionClasses/) returns each signal's default in
  `defaults`.

## Active series

[GetDataset](/obs-api/GetDataset/) returns `activeSeries`, an estimate of the distinct
metric series the dataset received in the previous complete hour:

```json
"activeSeries": { "from": "2026-10-09T13:00:00Z", "to": "2026-10-09T14:00:00Z", "estimate": 41250 }
```

The estimate is within about 1%, and counts by when points were received. It's `null` when the
estimate is briefly unavailable. `ezgh obs datasets get` shows it as **Active series**. A series
is one combination of metric, attributes, resource and scope. [Span metrics](/observability/span-metrics)
aren't counted.

## Access

| Action | Allows |
| --- | --- |
| `obs.metrics.ingest` | Sending metrics, with a project ingest key |
| `obs.metrics.query` | Querying a dataset's metrics with PromQL (returns metric data and label values) |

`ObsFullAccess` includes both. `ObsReadOnlyAccess` and `ReadOnlyAccess` include
`obs.metrics.query`, so read-only users can query metrics. A query reads only the datasets the caller may query, checked on every request.

## Trails

Requests to the Prometheus API aren't recorded in [Trails](/trails): queries, label and series
lookups, metadata and build info only read, and reads of telemetry aren't recorded. Sending
metrics isn't recorded either.

## Pricing

Metrics are metered by ingested bytes and by stored bytes, hot and cold. These meters aren't
billed yet.

Source: https://docs.ezghcloud.com/observability/metrics/index.mdx