Troubleshooting | EZGH Cloud

What each OTLP status and rejection means when sending logs, and what to change.

Errors follow the OTLP specification for each protocol. Over HTTP, the body is a google.rpc.Status, in the request’s encoding (protobuf or JSON); its message starts with the error code below (unauthenticated: …). Over gRPC, it’s the call’s status. Exporters log both.

Request errors

HTTP	gRPC	Code	Meaning	Retried by exporters	What to do
`401`	`UNAUTHENTICATED`	`unauthenticated`	No `authorization` header, a malformed one, an unknown or revoked ingest key, or another kind of credential (API key, CLI login)	No	Send `authorization: Bearer ezgh_ik_…` with a current ingest key. In `OTEL_EXPORTER_OTLP_HEADERS`, write the space as `%20`.
`403`	`PERMISSION_DENIED`	`access_denied`	The key doesn’t allow `obs.logs.ingest`, or the organization isn’t active (suspended or closed)	No	Create a key with `obs.logs.ingest`, or resolve the organization’s status in the console.
`429`	`RESOURCE_EXHAUSTED`, with retry info	`too_many_requests`	A rate limit: the organization’s or the dataset’s bytes or requests per second, the project’s 10 new environments an hour, or too many failed attempts with one key	Yes, after the server’s delay	Nothing for short bursts. If it persists, see Limits: lower the rate, or check that environment values aren’t unique per host or deploy.
`413`	`RESOURCE_EXHAUSTED`, without retry info	`payload_too_large`	The request is over 5 MiB compressed, or 64 MiB once decompressed	No	Batch by bytes with a smaller maximum: see the exporter settings.
`400`	`INVALID_ARGUMENT`	`invalid_request`	The body isn’t a valid OTLP logs request in the declared encoding, or `x-ezgh-placement` isn’t `hot` or `cold`, or is `cold` while cold storage isn’t offered yet	No	Check `Content-Type` and the placement header. Until cold storage is offered, send `hot` or leave the header out.
`415`	—	`unsupported_media_type`	OTLP/HTTP with a `Content-Type` other than `application/x-protobuf` or `application/json`, or a `Content-Encoding` other than `gzip`	No	Use the exporter’s defaults.
`404`	`UNIMPLEMENTED`	`not_found`	A signal that isn’t accepted yet: traces or metrics	No	Set `OTEL_TRACES_EXPORTER=none` and `OTEL_METRICS_EXPORTER=none`, or remove those pipelines.
`503`	`UNAVAILABLE`, with retry info	`unavailable`	Observability can’t accept data right now	Yes, after the server’s delay	Nothing: the exporter retries. Nothing from the refused request was stored.
`405`	—	`method_not_allowed`	OTLP/HTTP with a method other than `POST` (the answer’s `Allow` header says `POST`)	No	Send with `POST`, as exporters do.

A 200 (gRPC OK) means the request was stored durably, except any records listed in partial_success.

Records rejected in partial_success

Some records can be rejected while the rest of the request is stored. The response’s partial_success gives the number of rejected records, and a message that counts them by reason: rejected 3 log records: 2 with a body over 65536 bytes; 1 with environment "qa": its dataset is being deleted. Exporters log it and don’t retry: sending those records again gives the same answer.

The message says	Meaning	What to do
`event_time outside the accepted window`	The record’s event time is more than a day in the future, or older than its dataset’s retention	Check the sender’s clock and the timestamps it sets.
`a body over 65536 bytes`	The record’s body is over 64 KiB	Shorten it, or split it into several records.
`a map over 256 attributes`	An attribute map (resource, scope or record) has more than 256 entries	Send fewer attributes.
`a trace_id that isn't 16 bytes`, `a span_id that isn't 8 bytes`	A malformed trace or span ID	Fix the instrumentation that sets them.
`a record that, with its resource and scope, is too large for one segment`	The record, together with its resource and scope attributes, is too large to store, usually because of a very large attribute value	Shorten large attribute values.
`environment "<env>": the project's dataset limit is reached, so no dataset was created`	The environment has no dataset, and the project is at its dataset quota (`obs.datasets.perProject`)	Delete datasets you don’t need, or ask for a higher quota. See Service quotas.
`environment "<env>": its dataset is being deleted`	The environment’s dataset is being deleted	Wait until it’s gone; the next record creates a new dataset.
`environment "<env>": cold placement isn't allowed (its dataset's allowColdIngest is false)`	The sender asked for `x-ezgh-placement: cold`	Remove the header, or allow cold ingest on the dataset.
`environment "<env>": its retention class cold-365d has no hot tier, and cold storage isn't enabled (COLD_PATH_ENABLED)`	The dataset’s class stores everything in cold storage, which isn’t offered yet	Give the dataset another retention class until cold storage is offered.

The message can also carry warnings, even when nothing was rejected (rejected_log_records is 0): warning: ezgh.metadata.environment "prod/eu" isn't a valid value, so it was stored as "unknown". A string value is shown quoted, cut to 64 characters; a value that isn’t a string is shown by its kind (an int value, a double value, a bool value). Those records were stored, in the unknown dataset; fix the attribute’s value.

Records in the wrong dataset

They’re in unknown. Their environment was missing, empty, not a string, or not a valid value (letters, digits, ., _, -; 63 characters at most). Or it was set on the log record instead of the resource. Set ezgh.metadata.environment as a resource attribute; see service metadata.
Many small datasets appear. The environment value differs per host or deploy (a hostname or a version, for example). Set a fixed value. Delete the extra datasets with ezgh obs datasets delete.

Search finds nothing

Records are searchable a few seconds after they’re stored. The search answer’s visibleThrough says, for each part of the dataset, through when the accepted data was covered. ezgh obs search prints the oldest of those times: everything accepted before it was searched.
Search filters by event time. A sender with a wrong clock stores records outside the window you searched: widen --from and --to.
--environment and --application values are compared after trimming and lowercasing, as ingest stores them.

Search errors

Status	Code	Meaning	What to do
`422`	`query_too_expensive`	The search would read more than a search may	Narrow the time range or filters, or run an async query: `ezgh obs queries start`.
`400`	`cursor_expired`	A page’s cursor is read more than 10 minutes after the search’s first page	Search again without the cursor.
`400`	`invalid_cursor`	The cursor is from another search, dataset or caller, or the request changed the search’s window, filters or order	Search again without the cursor. A next page sends only the cursor and the limit.
`503`	`source_unavailable`	Part of the data can’t be read right now; no partial answer is returned	Retry after `Retry-After`.
`429`	`too_many_requests`	Too many searches at once in the organization	Retry after `Retry-After`.

Async query errors

Status	Code	Meaning	What to do
`429`	`too_many_requests`	The organization already has its most queries queued or running	Retry after `Retry-After`, or cancel a query you don’t need.
`409`	`query_not_finished`	Results were read before the query finished	Wait for it: `ezgh obs queries get <id> --wait`.
`409`	`query_not_succeeded`	The query failed or was cancelled, so it has no results	`ezgh obs queries get <id>` shows its error.
`410`	`results_expired`	The results’ time ran out (`resultsExpireAt`)	Run the query again.
`409`	`query_finished`	A finished query can’t be cancelled	Nothing to do.