Metrics

Observability stores the metrics your services send over OpenTelemetry (OTLP) or Prometheus remote write, next to their logs and traces. Metrics go to the dataset for their environment and are read with PromQL through a Prometheus-compatible API, so Grafana’s Prometheus data source and other Prometheus clients work unchanged.

Send OTLP metrics

Any OTLP metric exporter works: the OpenTelemetry Collector, SDKs and language agents. Set the base endpoint and the ingest key header; the exporter adds the metrics path itself.

OTEL_EXPORTER_OTLP_ENDPOINT=https://obs-ingest.ezghcloud.com
OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer%20ezgh_ik_…"
OTEL_METRICS_EXPORTER=otlp
OTEL_RESOURCE_ATTRIBUTES="service.name=checkout,ezgh.metadata.application=storefront,ezgh.metadata.environment=production"

This works with OTEL_EXPORTER_OTLP_PROTOCOL set to grpc, http/protobuf or http/json. OTEL_EXPORTER_OTLP_HEADERS uses the W3C Baggage format, so the space after Bearer is written %20.

The ingest key must allow obs.metrics.ingest. A key that doesn’t is refused with 403; create a key that allows it, or a second key:

ezgh ingest-keys create otel-collector --action obs.logs.ingest,obs.traces.ingest,obs.metrics.ingest --project web

• All five OTLP point types are stored: gauges, sums, histograms, exponential histograms and summaries, with their exemplars.
• Cumulative and delta temporality are both accepted and stored as sent. A sum or histogram with unspecified temporality is rejected.
• A point with the no recorded value flag is a Prometheus staleness marker: the series ends there.

With the Collector

The example uses the current exporter name, otlp_grpc. Collectors released before the rename call it otlp. A Collector’s prometheus receiver can scrape Prometheus targets into the same pipeline.

processors:
  resource:
    attributes:
      - { key: ezgh.metadata.environment, value: production, action: insert }
exporters:
  otlp_grpc/ezgh:
    endpoint: obs-ingest.ezghcloud.com:443
    headers: { authorization: "Bearer ${env:EZGH_INGEST_KEY}" }
service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [resource]
      exporters: [otlp_grpc/ezgh]

See Send from a Collector for batching, retries and routing to several projects.

Send with Prometheus remote write

A Prometheus server (or any remote-write sender) writes to POST https://obs-ingest.ezghcloud.com/api/v1/write, with the ingest key as a bearer token. Remote write 1.0 and 2.0 are both accepted, chosen by the request’s Content-Type.

global:
  external_labels:
    ezgh_environment: production
    ezgh_application: storefront
remote_write:
  - url: https://obs-ingest.ezghcloud.com/api/v1/write
    protobuf_message: io.prometheus.write.v2.Request
    authorization:
      credentials_file: /etc/prometheus/ezgh-ingest-key

• Environment and application come from the ezgh_environment and ezgh_application labels, normalized like the ezgh.metadata.* resource attributes (a missing or invalid value is unknown). They’re removed from the stored labels and read back as environment and application. Set them with external_labels, as above, or per target with relabeling.
• Use remote write 2.0 (protobuf_message: io.prometheus.write.v2.Request, Prometheus 3.0 and later) where you can: it carries each series’ type, unit and help. Remote write 1.0 samples are stored with the type unknown.
• Only bearer authorization is accepted; basic_auth and sigv4 are refused with 401.
• Classic histograms and summaries arrive as their _bucket, _sum, _count and {quantile} series and are stored that way. Native histograms (schema −4 to 8, and custom buckets) are read back as native histograms.

Remote write 2.0 answers also carry X-Prometheus-Remote-Write-Samples-Written, …-Histograms-Written and …-Exemplars-Written.

Environments and datasets

Each point goes to the dataset for its environment, created automatically the first time an environment sends data, as for logs and traces. A point without an environment goes to the unknown dataset. See Service metadata.

Query with PromQL

Each project has a Prometheus HTTP API at

https://obs.ezghcloud.com/v1/organizations/<organization ID>/projects/<project ID>/obs/prometheus

ezgh projects get <project> -o json shows both IDs. A query reads every dataset (environment) of the project that the caller may query, so one query can compare environments; select one with an environment matcher, such as up{environment="production"}.

• Requests and answers are Prometheus’s: POST bodies are form-encoded, times are RFC 3339 or Unix seconds, and answers are {"status": "success", "data": …}. Errors are {"status": "error", "errorType": …, "error": …} with an added code, the platform’s error code.
• Authenticate with Authorization: Bearer and an API key, a bot’s API key, or an OAuth access token. Ingest keys are refused.
• Without start, labels, label/<name>/values, series, metadata and query_exemplars cover the last 6 hours, not all time.
• The lookback delta is 5 minutes, as in Prometheus. A query’s lookback_delta can set 1 second to 1 hour.

Grafana

Add a Prometheus data source with the project’s base URL and a bot’s API key:

Or provision it:

apiVersion: 1
datasources:
  - name: EZGH web
    type: prometheus
    access: proxy
    url: https://obs.ezghcloud.com/v1/organizations/<organization ID>/projects/<project ID>/obs/prometheus
    jsonData:
      httpMethod: POST
      prometheusType: Prometheus
      httpHeaderName1: Authorization
    secureJsonData:
      httpHeaderValue1: Bearer ${EZGH_GRAFANA_KEY}

One data source reads one project. Grafana alerting works against it like any Prometheus.

From the CLI

ezgh obs metrics query 'sum by (job) (rate(http_server_request_duration_seconds_count[5m]))' --project web
ezgh obs metrics query 'up{environment="production"}' --from 6h --step 5m --project web
ezgh obs metrics label-values __name__ --environment production --project web

See ezgh obs metrics. -o json prints the API’s answer.

Names and labels

OTLP metric names and attributes are translated to Prometheus names when they’re read, the way Prometheus’s own OTLP receiver translates them by default. What was sent is stored unchanged.

• Characters other than letters, digits and _ become _; the unit becomes a suffix (annotations in braces are dropped), and a monotonic sum gets _total.

• Labels are the point’s attributes, translated the same way (http.response.status_code is http_response_status_code). Two attributes that translate to the same label have their values joined with ;.

• Every series also has environment and application (from ezgh.metadata.*), job (service.namespace/service.name, or service.name) and instance (service.instance.id). A sent attribute or remote-write label named environment or application with a different value is read as exported_environment or exported_application, so environment always names the dataset.

• Other resource attributes are in target_info: one series per resource, value 1, with the resource attributes as labels plus environment, application, job and instance. Join it to add them:

  sum by (k8s_namespace_name) (
    rate(http_server_request_duration_seconds_count[5m])
    * on (environment, job, instance) group_left (k8s_namespace_name) target_info
  )

• The instrumentation scope’s name and version aren’t labels. Series that differ only in scope or in a resource attribute other than job and instance read as one series; the answer then carries a warning that series were merged. EZGH Obs Agent 0.3.0 copies the Kubernetes resource attributes onto each point, so its series have k8s_cluster_name, k8s_node_name, k8s_pod_name and the other Kubernetes labels.

• Delta sums and histograms are accumulated from the start of the query, so rate, increase and histogram_quantile are exact. A delta counter’s raw value counts from the query’s start.

• Remote-write series keep their names and labels as sent, apart from ezgh_environment and ezgh_application.

• Generated series. Every dataset that receives traces also has the traces_spanmetrics_* and traces_service_graph_* series that Observability derives from its spans. They’re free; see Span metrics.

Limits

A point that breaks a limit is rejected; the rest of the request is stored. OTLP answers say how many points were rejected and why in partial_success; remote write answers 400.

Counts at or above 2⁶³ are rejected. From remote write, gauge native histograms and float histograms with fractional counts are rejected.

Query budgets

Every query reads stored points, so a long range at a short step can be expensive. A query over a budget is refused with 422 and code query_too_expensive, with budget naming which one, never cut short. Narrow it: a shorter range, a larger step, or more label matchers.

Retention

Each signal of a dataset has its own retention class. Metrics default to 14d-395d, 13 months, so a year can be compared with the last; logs and traces default to 7d-30d. Every class can be used for every signal.

• A dataset’s retention holds each signal’s class: {"logs": "7d-30d", "traces": "7d-30d", "metrics": "14d-395d"}. retentionClass is the logs class, which traces follow unless set separately; changing it never changes the metrics class.
• Set one signal’s class with retention in CreateDataset or UpdateDataset, such as {"retention": {"metrics": "14d-365d"}}. Where this isn’t available yet, the request is refused with 409 signal_retention_unavailable.
• A new class applies to points that arrive afterward; stored points keep the class they arrived with.
• Points are kept at full resolution for the whole class: there’s no downsampling.
• ezgh obs datasets get shows every signal’s class, and ListRetentionClasses returns each signal’s default in defaults.

Active series

GetDataset returns activeSeries, an estimate of the distinct metric series the dataset received in the previous complete hour:

"activeSeries": { "from": "2026-10-09T13:00:00Z", "to": "2026-10-09T14:00:00Z", "estimate": 41250 }

The estimate is within about 1%, and counts by when points were received. It’s null when the estimate is briefly unavailable. ezgh obs datasets get shows it as Active series. A series is one combination of metric, attributes, resource and scope. Span metrics aren’t counted.

Access

ObsFullAccess includes both. ObsReadOnlyAccess and ReadOnlyAccess include obs.metrics.query, so read-only users can query metrics. A query reads only the datasets the caller may query, checked on every request.

Trails

Requests to the Prometheus API aren’t recorded in Trails: queries, label and series lookups, metadata and build info only read, and reads of telemetry aren’t recorded. Sending metrics isn’t recorded either.

Pricing

Metrics are metered by ingested bytes and by stored bytes, hot and cold. These meters aren’t billed yet.