---
title: "Send from a Collector"
description: "OpenTelemetry Collector configurations for Kubernetes agents and gateways, service metadata, batching and several projects."
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.ezghcloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Send from a Collector

Use the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) contrib distribution
(`otel/opentelemetry-collector-contrib`) to collect logs from hosts and Kubernetes nodes, or as a
gateway in front of your services. The examples use current component names: the OTLP exporters
are `otlp_grpc` and `otlp_http`. Collectors released before the rename call them `otlp` and
`otlphttp`.

Put the ingest key in the Collector's environment as `EZGH_INGEST_KEY`, from a Kubernetes secret
or your secret manager. In the Collector's YAML, the header is written with a plain space:
`Bearer ${env:EZGH_INGEST_KEY}`.

## The exporter

```yaml
exporters:
  otlp_grpc/ezgh:
    endpoint: obs-ingest.ezghcloud.com:443
    headers:
      authorization: "Bearer ${env:EZGH_INGEST_KEY}"
    sending_queue:
      batch:
        sizer: bytes
        min_size: 1048576      # send at 1 MiB…
        max_size: 4194304      # …never more than 4 MiB a request
        flush_timeout: 5s
```

- **OTLP/HTTP** works the same: `otlp_http/ezgh` with `endpoint: https://obs-ingest.ezghcloud.com`.
  The exporter adds `/v1/logs`.
- **Batching by bytes** keeps each request well under the 5 MiB compressed request limit (see
  [Limits](/observability/limits)). The exporters compress with gzip by default.
- **Retries:** the exporter retries rate limits and unavailability, honoring the server's retry
  delay. It doesn't retry requests that are too large or not authorized.

## Service metadata with the resource processor

Records are routed by the resource attribute `ezgh.metadata.environment`. For workloads that
don't set it, add it in the Collector. `insert` adds the attribute only where it's missing, so
services that set their own keep it. `upsert` would replace it.

```yaml
processors:
  resource:
    attributes:
      - key: ezgh.metadata.environment
        value: production
        action: insert
      - key: ezgh.metadata.application
        value: storefront
        action: insert
```

## Kubernetes: an agent on every node

[EZGH Obs Agent](/observability/kubernetes-agent) installs a configured Collector with Helm,
including cluster state, events and the Kubernetes labels the
[Kubernetes](/observability/kubernetes) section needs. To run your own instead, a
DaemonSet Collector reads container logs from each node, adds Kubernetes metadata, and sends
them. This configuration takes each pod's application from its `app.kubernetes.io/part-of` label
and sets one environment for the cluster. It uses Collector Contrib's current component names; before
0.146 and 0.149, `k8s_attributes` and `file_log` were called `k8sattributes` and `filelog`:

```yaml
receivers:
  file_log:
    include: [/var/log/pods/*/*/*.log]
    exclude: [/var/log/pods/*/otel-collector*/*.log]
    start_at: end
    include_file_path: true
    operators:
      - type: container
        id: container-parser

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  k8s_attributes:
    pod_association:
      - sources:
          - from: resource_attribute
            name: k8s.pod.uid
    extract:
      metadata: [k8s.namespace.name, k8s.pod.name, k8s.deployment.name, k8s.container.name]
      labels:
        - tag_name: ezgh.metadata.application
          key: app.kubernetes.io/part-of
          from: pod
  resource:
    attributes:
      - key: ezgh.metadata.environment
        value: production
        action: insert

exporters:
  otlp_grpc/ezgh:
    endpoint: obs-ingest.ezghcloud.com:443
    headers:
      authorization: "Bearer ${env:EZGH_INGEST_KEY}"
    sending_queue:
      batch:
        sizer: bytes
        min_size: 1048576
        max_size: 4194304
        flush_timeout: 5s

service:
  pipelines:
    logs:
      receivers: [file_log]
      processors: [memory_limiter, k8s_attributes, resource]
      exporters: [otlp_grpc/ezgh]
```

The DaemonSet mounts `/var/log/pods` read-only. Its service account may get, list and watch pods,
namespaces and ReplicaSets, which `k8s_attributes` needs. The
[OpenTelemetry Collector Helm chart](https://github.com/open-telemetry/opentelemetry-helm-charts)
sets these up with `mode: daemonset` and its `logsCollection` and `kubernetesAttributes` presets.
Put the `resource` processor and the exporter above in its `config`. For metrics, add the
[`transform/k8slabels` processor](/observability/kubernetes-agent#in-your-own-collector) so PromQL
can tell nodes, pods and containers apart.

## A gateway

Services send OTLP to a gateway Collector in your network, which sends everything on. One
gateway can carry many services and environments: each record still goes to its own
environment's dataset.

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25

exporters:
  otlp_grpc/ezgh:
    endpoint: obs-ingest.ezghcloud.com:443
    headers:
      authorization: "Bearer ${env:EZGH_INGEST_KEY}"
    sending_queue:
      batch:
        sizer: bytes
        min_size: 1048576
        max_size: 4194304
        flush_timeout: 5s

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [otlp_grpc/ezgh]
```

## Several projects from one Collector

An ingest key sends to one project. A Collector that serves several projects has one exporter per
project, each with its own key, and routes records to them with the `routing` connector. This
example routes by Kubernetes namespace: `payments` goes to one project and everything else to
another.

```yaml
connectors:
  routing:
    default_pipelines: [logs/web]
    table:
      - condition: resource.attributes["k8s.namespace.name"] == "payments"
        pipelines: [logs/payments]

exporters:
  otlp_grpc/web:
    endpoint: obs-ingest.ezghcloud.com:443
    headers:
      authorization: "Bearer ${env:EZGH_INGEST_KEY_WEB}"
    sending_queue:
      batch: { sizer: bytes, min_size: 1048576, max_size: 4194304, flush_timeout: 5s }
  otlp_grpc/payments:
    endpoint: obs-ingest.ezghcloud.com:443
    headers:
      authorization: "Bearer ${env:EZGH_INGEST_KEY_PAYMENTS}"
    sending_queue:
      batch: { sizer: bytes, min_size: 1048576, max_size: 4194304, flush_timeout: 5s }

service:
  pipelines:
    logs/in:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [routing]
    logs/web:
      receivers: [routing]
      exporters: [otlp_grpc/web]
    logs/payments:
      receivers: [routing]
      exporters: [otlp_grpc/payments]
```

Add the `otlp` receiver and `memory_limiter` processor from the gateway example. Use
`otlp_http` exporters the same way.

## Sending to cold storage

To write a pipeline's records straight to cold storage (for a dataset that allows it), add the
header to its exporter:

```yaml
    headers:
      authorization: "Bearer ${env:EZGH_INGEST_KEY}"
      x-ezgh-placement: cold
```

Source: https://docs.ezghcloud.com/observability/send-with-collector/index.mdx