---
title: "Observability"
description: "Send logs, traces and metrics with OpenTelemetry, keep them by environment, search and query them, see your services and their dependencies, and build dashboards."
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.ezghcloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability

Observability keeps the logs, traces and metrics your services send with
[OpenTelemetry](https://opentelemetry.io), and lets you search logs and traces, query metrics
with PromQL, see every service with its requests, errors, latency and dependencies, and chart all
three on dashboards. Any OpenTelemetry SDK, language agent or Collector can send them over OTLP/gRPC or
OTLP/HTTP, with no EZGH-specific code: you set an endpoint, a header and two resource attributes.
Prometheus servers can send metrics with remote write.

- [Ingest keys](/observability/ingest-keys) — Create the credential your services send logs with.
- [Send from an SDK](/observability/send-with-sdks) — Environment variables for Java, Python, Node.js, Go and .NET.
- [Send from a Collector](/observability/send-with-collector) — Agent and gateway configurations, and routing to several projects.
- [Kubernetes agent](/observability/kubernetes-agent) — Install EZGH Obs Agent with Helm for a cluster's logs, metrics, cluster state and events.
- [Troubleshooting](/observability/troubleshooting) — What each error means, and what to change.
- [Limits](/observability/limits) — Request and record sizes, rates, retention and search.
- [Traces](/observability/traces) — Send OpenTelemetry traces, search spans and read whole traces.
- [Trace sampling](/observability/sampling) — Keep errors, slow requests and a share of the rest; store and pay for only what's kept.
- [Metrics](/observability/metrics) — Send OTLP metrics or Prometheus remote write, and query them with PromQL and Grafana.
- [Dashboards](/observability/dashboards) — Chart logs, traces and metrics on saved, shareable dashboards, and manage them as JSON.
- [Kubernetes](/observability/kubernetes) — Your clusters, nodes, workloads and pods, with their problems, events and agent health.
- [Service catalog](/observability/service-catalog) — Every service sending telemetry, discovered automatically, with its stats, versions and dependencies.
- [Service map](/observability/service-map) — Your services and the databases, queues and hosts they call, from paired spans.
- [Span metrics](/observability/span-metrics) — The free request, error and latency series generated from your traces, for PromQL and Grafana.

## Concepts

| Term | Meaning |
| --- | --- |
| Ingest key | A project's credential for sending telemetry: `ezgh_ik_` followed by 43 letters and digits. It has an ID, an `igk_` slug and a resource name like any other resource, and can only send data to its own project. See [Ingest keys](/observability/ingest-keys). |
| Dataset | One **environment** of a project, such as `production` or `staging`. A dataset has an ID, a `dst_` slug, a resource name, tags, a retention class and a default placement. |
| Application | The service or system a record comes from, as you name it, such as `storefront`. Stored with every record and used as a search filter. |
| Retention class | How long a dataset keeps records, counted from each record's event time: days in hot storage, and days in all. |
| Placement | Where a record is stored first: `hot` (searched fastest) or `cold`. Records move from hot to cold storage when their hot days end, and stay searchable. |

The management API is at `https://obs.ezghcloud.com`, and logs are sent to
`https://obs-ingest.ezghcloud.com`. The `ezgh obs` and `ezgh ingest-keys` commands cover the
same tasks as the management API.

## Service metadata

Every service sets two **resource attributes**, alongside `service.name`:

| Attribute | Example | Used for |
| --- | --- | --- |
| `ezgh.metadata.environment` | `production` | Choosing the record's dataset |
| `ezgh.metadata.application` | `storefront` | A stored field you search and count by |

```sh
OTEL_RESOURCE_ATTRIBUTES="service.name=checkout,ezgh.metadata.application=storefront,ezgh.metadata.environment=production"
```

- Values are trimmed and lowercased. They must then be 1 to 63 characters: letters, digits, `.`,
  `_` and `-`, starting with a letter or digit. `Production ` and `production` are the same
  environment.
- A missing, empty, non-string or invalid value is stored as `unknown`. Records with no
  environment go to a dataset named `unknown`.
- Only **resource** attributes count. The same keys set on a log record or an instrumentation
  scope are kept as ordinary attributes and don't choose a dataset.
- Records keep the attributes exactly as they were sent.

For workloads you can't change, a Collector adds the attributes with its `resource` processor; see
[Send from a Collector](/observability/send-with-collector).

## Datasets are environments

Each record goes to the dataset for its project (from the ingest key) and its environment.

- **Created on first use:** the first record from a new environment creates its dataset. The
  dataset gets retention class `7d-30d` (`14d-395d` for metrics) and hot placement.
- **Created ahead:** create a dataset yourself to choose its retention class before data arrives.

### CLI

```sh
ezgh obs datasets create production --retention-class 14d-365d --project web
ezgh obs datasets list --project web
```
### API

Call [CreateDataset](/obs-api/CreateDataset/). [ListDatasets](/obs-api/ListDatasets/)
lists a project's datasets.

- One request can carry several environments: a gateway Collector that batches many services
  sends each record to its own environment's dataset.
- A project can start at most 10 new environments an hour, and has a quota of datasets; see
  [Limits](/observability/limits). An environment that already has a dataset can't get another
  (`409 environment_taken`).
- Changing a dataset's retention class ([UpdateDataset](/obs-api/UpdateDataset/),
  `ezgh obs datasets update`) applies to records that arrive afterward. Stored records keep the
  class they arrived with.
- Deleting a dataset ([DeleteDataset](/obs-api/DeleteDataset/), `ezgh obs datasets delete`)
  removes all its records in the background. Its status is `deleting` until then, and records for
  its environment are rejected. After that, the environment's next record creates a new, empty
  dataset. To stop data for good, revoke the ingest key.

## Retention classes

| Class | Hot days | Total days |
| --- | --- | --- |
| `7d-30d` | 7 | 30 |
| `7d-90d` | 7 | 90 |
| `14d-365d` | 14 | 365 |
| `14d-395d` | 14 | 395 |
| `cold-365d` | 0 | 365 |

Days count from each record's event time. Each signal of a dataset has its own class: logs and
traces default to `7d-30d`, metrics to `14d-395d`; see [Metrics](/observability/metrics#retention). `cold-365d` stores every record in cold storage:
datasets with it get cold placement and cold ingest automatically. [ListRetentionClasses](/obs-api/ListRetentionClasses/)
and `ezgh obs retention-classes list` list the classes.

## Sending to cold storage

A sender can ask for cold storage with the header `x-ezgh-placement: cold` (gRPC metadata of the
same name), if the dataset allows it (`allowColdIngest`). `hot` asks for hot storage. Without the
header, the dataset's default placement applies.

## When data is searchable

A `200` (or gRPC `OK`) means Observability stored the request durably. Records are searchable a
few seconds later. Search answers say which accepted data they cover (`visibleThrough`), so an
empty answer for the last few seconds isn't proof that nothing arrived.

Delivery is at least once. If an exporter retries a request whose answer it didn't receive, the
records can be stored twice.

## Search and queries

- **Search** reads one dataset, synchronously: `ezgh obs search` or
  [SearchLogs](/obs-api/SearchLogs/). It filters by time, severity, service, application,
  environment, trace ID, attributes and body text, within a time and size budget.
- **Async queries** read 1 to 10 datasets of a project in the background, under a larger
  budget, and can count records by application, environment, severity or service:
  `ezgh obs queries start` or [StartQuery](/obs-api/StartQuery/). Their results are kept for a
  day by default.

## Access

Access is policy-based, like the rest of EZGH Cloud. The `obs.*` actions control datasets,
search, queries, dashboards and the service catalog. Reading telemetry is a read: `obs.logs.search` (search and the
console's log reads), `obs.traces.search`, `obs.traces.getSpans` and `obs.metrics.query` are in
`ObsFullAccess`, `ObsReadOnlyAccess` and `ReadOnlyAccess`, so read-only users can read logs,
traces and metrics. Async queries are granted like writes, because starting one creates a job:
`obs.queries.start`, `obs.queries.cancel` and `obs.queries.getResults` are only in
`ObsFullAccess`. `obs.services.list` and `obs.services.get` view the
[service catalog](/observability/service-catalog), and are in all three policies. To keep someone from reading telemetry, give them a custom policy without the
read actions. Ingest keys are managed with the project actions `ingestKeys.create`,
`ingestKeys.list`, `ingestKeys.get` and `ingestKeys.delete`.

## Trails

Management calls are recorded in [Trails](/trails) with `eventSource` `obs.ezghcloud.com`,
including datasets created on first use (`CreateDataset`, with the ingest key's ID as `userIdentity`),
services the catalog adds (`CreateService`, likewise), dashboards and async queries. An async query's filters are recorded as hidden values, because they
can contain your data. Sending telemetry isn't recorded, and neither is reading it: searches, the
console's log and trace views, dashboard panels and PromQL queries aren't in Trails. A dashboard's
[version history](/observability/dashboards#version-history) shows which queries it held.

Source: https://docs.ezghcloud.com/observability/index.mdx