Observability

Observability keeps the logs, traces and metrics your services send with OpenTelemetry, and lets you search logs and traces, query metrics with PromQL, see every service with its requests, errors, latency and dependencies, and chart all three on dashboards. Any OpenTelemetry SDK, language agent or Collector can send them over OTLP/gRPC or OTLP/HTTP, with no EZGH-specific code: you set an endpoint, a header and two resource attributes. Prometheus servers can send metrics with remote write.

Concepts

The management API is at https://obs.ezghcloud.com, and logs are sent to https://obs-ingest.ezghcloud.com. The ezgh obs and ezgh ingest-keys commands cover the same tasks as the management API.

Service metadata

Every service sets two resource attributes, alongside service.name:

OTEL_RESOURCE_ATTRIBUTES="service.name=checkout,ezgh.metadata.application=storefront,ezgh.metadata.environment=production"

• Values are trimmed and lowercased. They must then be 1 to 63 characters: letters, digits, ., _ and -, starting with a letter or digit. Production and production are the same environment.
• A missing, empty, non-string or invalid value is stored as unknown. Records with no environment go to a dataset named unknown.
• Only resource attributes count. The same keys set on a log record or an instrumentation scope are kept as ordinary attributes and don’t choose a dataset.
• Records keep the attributes exactly as they were sent.

For workloads you can’t change, a Collector adds the attributes with its resource processor; see Send from a Collector.

Datasets are environments

Each record goes to the dataset for its project (from the ingest key) and its environment.

• Created on first use: the first record from a new environment creates its dataset. The dataset gets retention class 7d-30d (14d-395d for metrics) and hot placement.
• Created ahead: create a dataset yourself to choose its retention class before data arrives.

ezgh obs datasets create production --retention-class 14d-365d --project web
ezgh obs datasets list --project web

• One request can carry several environments: a gateway Collector that batches many services sends each record to its own environment’s dataset.
• A project can start at most 10 new environments an hour, and has a quota of datasets; see Limits. An environment that already has a dataset can’t get another (409 environment_taken).
• Changing a dataset’s retention class (UpdateDataset, ezgh obs datasets update) applies to records that arrive afterward. Stored records keep the class they arrived with.
• Deleting a dataset (DeleteDataset, ezgh obs datasets delete) removes all its records in the background. Its status is deleting until then, and records for its environment are rejected. After that, the environment’s next record creates a new, empty dataset. To stop data for good, revoke the ingest key.

Retention classes

Days count from each record’s event time. Each signal of a dataset has its own class: logs and traces default to 7d-30d, metrics to 14d-395d; see Metrics. cold-365d stores every record in cold storage: datasets with it get cold placement and cold ingest automatically. ListRetentionClasses and ezgh obs retention-classes list list the classes.

Sending to cold storage

A sender can ask for cold storage with the header x-ezgh-placement: cold (gRPC metadata of the same name), if the dataset allows it (allowColdIngest). hot asks for hot storage. Without the header, the dataset’s default placement applies.

When data is searchable

A 200 (or gRPC OK) means Observability stored the request durably. Records are searchable a few seconds later. Search answers say which accepted data they cover (visibleThrough), so an empty answer for the last few seconds isn’t proof that nothing arrived.

Delivery is at least once. If an exporter retries a request whose answer it didn’t receive, the records can be stored twice.

Search and queries

• Search reads one dataset, synchronously: ezgh obs search or SearchLogs. It filters by time, severity, service, application, environment, trace ID, attributes and body text, within a time and size budget.
• Async queries read 1 to 10 datasets of a project in the background, under a larger budget, and can count records by application, environment, severity or service: ezgh obs queries start or StartQuery. Their results are kept for a day by default.

Access

Access is policy-based, like the rest of EZGH Cloud. The obs.* actions control datasets, search, queries, dashboards and the service catalog. Reading telemetry is a read: obs.logs.search (search and the console’s log reads), obs.traces.search, obs.traces.getSpans and obs.metrics.query are in ObsFullAccess, ObsReadOnlyAccess and ReadOnlyAccess, so read-only users can read logs, traces and metrics. Async queries are granted like writes, because starting one creates a job: obs.queries.start, obs.queries.cancel and obs.queries.getResults are only in ObsFullAccess. obs.services.list and obs.services.get view the service catalog, and are in all three policies. To keep someone from reading telemetry, give them a custom policy without the read actions. Ingest keys are managed with the project actions ingestKeys.create, ingestKeys.list, ingestKeys.get and ingestKeys.delete.

Trails

Management calls are recorded in Trails with eventSource obs.ezghcloud.com, including datasets created on first use (CreateDataset, with the ingest key’s ID as userIdentity), services the catalog adds (CreateService, likewise), dashboards and async queries. An async query’s filters are recorded as hidden values, because they can contain your data. Sending telemetry isn’t recorded, and neither is reading it: searches, the console’s log and trace views, dashboard panels and PromQL queries aren’t in Trails. A dashboard’s version history shows which queries it held.