---
title: "Troubleshooting"
description: "What each OTLP status and rejection means when sending logs, and what to change."
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.ezghcloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

Errors follow the OTLP specification for each protocol. Over HTTP, the body is a
`google.rpc.Status`, in the request's encoding (protobuf or JSON); its `message` starts with the
error code below (`unauthenticated: …`). Over gRPC, it's the call's status. Exporters log both.

## Request errors

| HTTP | gRPC | Code | Meaning | Retried by exporters | What to do |
| --- | --- | --- | --- | --- | --- |
| `401` | `UNAUTHENTICATED` | `unauthenticated` | No `authorization` header, a malformed one, an unknown or revoked ingest key, or another kind of credential (API key, CLI login) | No | Send `authorization: Bearer ezgh_ik_…` with a current ingest key. In `OTEL_EXPORTER_OTLP_HEADERS`, write the space as `%20`. |
| `403` | `PERMISSION_DENIED` | `access_denied` | The key doesn't allow `obs.logs.ingest`, or the organization isn't active (suspended or closed) | No | Create a key with `obs.logs.ingest`, or resolve the organization's status in the console. |
| `429` | `RESOURCE_EXHAUSTED`, with retry info | `too_many_requests` | A rate limit: the organization's or the dataset's bytes or requests per second, the project's 10 new environments an hour, or too many failed attempts with one key | Yes, after the server's delay | Nothing for short bursts. If it persists, see [Limits](/observability/limits): lower the rate, or check that environment values aren't unique per host or deploy. |
| `413` | `RESOURCE_EXHAUSTED`, without retry info | `payload_too_large` | The request is over 5 MiB compressed, or 64 MiB once decompressed | No | Batch by bytes with a smaller maximum: see [the exporter settings](/observability/send-with-collector#the-exporter). |
| `400` | `INVALID_ARGUMENT` | `invalid_request` | The body isn't a valid OTLP logs request in the declared encoding, or `x-ezgh-placement` isn't `hot` or `cold`, or is `cold` while cold storage isn't offered yet | No | Check `Content-Type` and the placement header. Until cold storage is offered, send `hot` or leave the header out. |
| `415` | — | `unsupported_media_type` | OTLP/HTTP with a `Content-Type` other than `application/x-protobuf` or `application/json`, or a `Content-Encoding` other than `gzip` | No | Use the exporter's defaults. |
| `404` | `UNIMPLEMENTED` | `not_found` | A signal that isn't accepted yet: traces or metrics | No | Set `OTEL_TRACES_EXPORTER=none` and `OTEL_METRICS_EXPORTER=none`, or remove those pipelines. |
| `503` | `UNAVAILABLE`, with retry info | `unavailable` | Observability can't accept data right now | Yes, after the server's delay | Nothing: the exporter retries. Nothing from the refused request was stored. |
| `405` | — | `method_not_allowed` | OTLP/HTTP with a method other than `POST` (the answer's `Allow` header says `POST`) | No | Send with `POST`, as exporters do. |

A `200` (gRPC `OK`) means the request was stored durably, except any records listed in
`partial_success`.

## Records rejected in partial_success

Some records can be rejected while the rest of the request is stored. The response's
`partial_success` gives the number of rejected records, and a message that counts them by reason:
`rejected 3 log records: 2 with a body over 65536 bytes; 1 with environment "qa": its dataset is
being deleted`. Exporters log it and don't retry: sending those records again gives the same
answer.

| The message says | Meaning | What to do |
| --- | --- | --- |
| `event_time outside the accepted window` | The record's event time is more than a day in the future, or older than its dataset's retention | Check the sender's clock and the timestamps it sets. |
| `a body over 65536 bytes` | The record's body is over 64 KiB | Shorten it, or split it into several records. |
| `a map over 256 attributes` | An attribute map (resource, scope or record) has more than 256 entries | Send fewer attributes. |
| `a trace_id that isn't 16 bytes`, `a span_id that isn't 8 bytes` | A malformed trace or span ID | Fix the instrumentation that sets them. |
| `a record that, with its resource and scope, is too large for one segment` | The record, together with its resource and scope attributes, is too large to store, usually because of a very large attribute value | Shorten large attribute values. |
| `environment "<env>": the project's dataset limit is reached, so no dataset was created` | The environment has no dataset, and the project is at its dataset quota (`obs.datasets.perProject`) | Delete datasets you don't need, or ask for a higher quota. See [Service quotas](/quotas). |
| `environment "<env>": its dataset is being deleted` | The environment's dataset is being deleted | Wait until it's gone; the next record creates a new dataset. |
| `environment "<env>": cold placement isn't allowed (its dataset's allowColdIngest is false)` | The sender asked for `x-ezgh-placement: cold` | Remove the header, or allow cold ingest on the dataset. |
| `environment "<env>": its retention class cold-365d has no hot tier, and cold storage isn't enabled (COLD_PATH_ENABLED)` | The dataset's class stores everything in cold storage, which isn't offered yet | Give the dataset another retention class until cold storage is offered. |

The message can also carry warnings, even when nothing was rejected (`rejected_log_records` is
`0`): `warning: ezgh.metadata.environment "prod/eu" isn't a valid value, so it was stored as
"unknown"`. A string value is shown quoted, cut to 64 characters; a value that isn't a string is
shown by its kind (`an int value`, `a double value`, `a bool value`). Those records were stored,
in the `unknown` dataset; fix the attribute's value.

## Records in the wrong dataset

- **They're in `unknown`.** Their environment was missing, empty, not a string, or not a valid
  value (letters, digits, `.`, `_`, `-`; 63 characters at most). Or it was set on the log record
  instead of the resource. Set `ezgh.metadata.environment` as a resource attribute; see
  [service metadata](/observability#service-metadata).
- **Many small datasets appear.** The environment value differs per host or deploy (a hostname or
  a version, for example). Set a fixed value. Delete the extra datasets with
  `ezgh obs datasets delete`.

## Search finds nothing

- Records are searchable a few seconds after they're stored. The search answer's
  `visibleThrough` says, for each part of the dataset, through when the accepted data was
  covered. `ezgh obs search` prints the oldest of those times: everything accepted before it was
  searched.
- Search filters by **event time**. A sender with a wrong clock stores records outside the window
  you searched: widen `--from` and `--to`.
- `--environment` and `--application` values are compared after trimming and lowercasing, as
  ingest stores them.

## Search errors

| Status | Code | Meaning | What to do |
| --- | --- | --- | --- |
| `422` | `query_too_expensive` | The search would read more than a search may | Narrow the time range or filters, or run an async query: `ezgh obs queries start`. |
| `400` | `cursor_expired` | A page's cursor is read more than 10 minutes after the search's first page | Search again without the cursor. |
| `400` | `invalid_cursor` | The cursor is from another search, dataset or caller, or the request changed the search's window, filters or order | Search again without the cursor. A next page sends only the cursor and the limit. |
| `503` | `source_unavailable` | Part of the data can't be read right now; no partial answer is returned | Retry after `Retry-After`. |
| `429` | `too_many_requests` | Too many searches at once in the organization | Retry after `Retry-After`. |

## Async query errors

| Status | Code | Meaning | What to do |
| --- | --- | --- | --- |
| `429` | `too_many_requests` | The organization already has its most queries queued or running | Retry after `Retry-After`, or cancel a query you don't need. |
| `409` | `query_not_finished` | Results were read before the query finished | Wait for it: `ezgh obs queries get <id> --wait`. |
| `409` | `query_not_succeeded` | The query failed or was cancelled, so it has no results | `ezgh obs queries get <id>` shows its error. |
| `410` | `results_expired` | The results' time ran out (`resultsExpireAt`) | Run the query again. |
| `409` | `query_finished` | A finished query can't be cancelled | Nothing to do. |

Source: https://docs.ezghcloud.com/observability/troubleshooting/index.mdx