Send from a Collector

OpenTelemetry Collector configurations for Kubernetes agents and gateways, service metadata, batching and several projects.

Use the OpenTelemetry Collector contrib distribution (otel/opentelemetry-collector-contrib) to collect logs from hosts and Kubernetes nodes, or as a gateway in front of your services. The examples use current component names: the OTLP exporters are otlp_grpc and otlp_http. Collectors released before the rename call them otlp and otlphttp.

Put the ingest key in the Collector’s environment as EZGH_INGEST_KEY, from a Kubernetes secret or your secret manager. In the Collector’s YAML, the header is written with a plain space: Bearer ${env:EZGH_INGEST_KEY}.

The exporter

exporters:
  otlp_grpc/ezgh:
    endpoint: obs-ingest.ezghcloud.com:443
    headers:
      authorization: "Bearer ${env:EZGH_INGEST_KEY}"
    sending_queue:
      batch:
        sizer: bytes
        min_size: 1048576      # send at 1 MiB…
        max_size: 4194304      # …never more than 4 MiB a request
        flush_timeout: 5s

OTLP/HTTP works the same: otlp_http/ezgh with endpoint: https://obs-ingest.ezghcloud.com. The exporter adds /v1/logs.
Batching by bytes keeps each request well under the 5 MiB compressed request limit (see Limits). The exporters compress with gzip by default.
Retries: the exporter retries rate limits and unavailability, honoring the server’s retry delay. It doesn’t retry requests that are too large or not authorized.

Service metadata with the resource processor

Records are routed by the resource attribute ezgh.metadata.environment. For workloads that don’t set it, add it in the Collector. insert adds the attribute only where it’s missing, so services that set their own keep it. upsert would replace it.

processors:
  resource:
    attributes:
      - key: ezgh.metadata.environment
        value: production
        action: insert
      - key: ezgh.metadata.application
        value: storefront
        action: insert

Kubernetes: an agent on every node

EZGH Obs Agent installs a configured Collector with Helm, including cluster state, events and the Kubernetes labels the Kubernetes section needs. To run your own instead, a DaemonSet Collector reads container logs from each node, adds Kubernetes metadata, and sends them. This configuration takes each pod’s application from its app.kubernetes.io/part-of label and sets one environment for the cluster. It uses Collector Contrib’s current component names; before 0.146 and 0.149, k8s_attributes and file_log were called k8sattributes and filelog:

receivers:
  file_log:
    include: [/var/log/pods/*/*/*.log]
    exclude: [/var/log/pods/*/otel-collector*/*.log]
    start_at: end
    include_file_path: true
    operators:
      - type: container
        id: container-parser

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  k8s_attributes:
    pod_association:
      - sources:
          - from: resource_attribute
            name: k8s.pod.uid
    extract:
      metadata: [k8s.namespace.name, k8s.pod.name, k8s.deployment.name, k8s.container.name]
      labels:
        - tag_name: ezgh.metadata.application
          key: app.kubernetes.io/part-of
          from: pod
  resource:
    attributes:
      - key: ezgh.metadata.environment
        value: production
        action: insert

exporters:
  otlp_grpc/ezgh:
    endpoint: obs-ingest.ezghcloud.com:443
    headers:
      authorization: "Bearer ${env:EZGH_INGEST_KEY}"
    sending_queue:
      batch:
        sizer: bytes
        min_size: 1048576
        max_size: 4194304
        flush_timeout: 5s

service:
  pipelines:
    logs:
      receivers: [file_log]
      processors: [memory_limiter, k8s_attributes, resource]
      exporters: [otlp_grpc/ezgh]

The DaemonSet mounts /var/log/pods read-only. Its service account may get, list and watch pods, namespaces and ReplicaSets, which k8s_attributes needs. The OpenTelemetry Collector Helm chart sets these up with mode: daemonset and its logsCollection and kubernetesAttributes presets. Put the resource processor and the exporter above in its config. For metrics, add the transform/k8slabels processor so PromQL can tell nodes, pods and containers apart.

A gateway

Services send OTLP to a gateway Collector in your network, which sends everything on. One gateway can carry many services and environments: each record still goes to its own environment’s dataset.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25

exporters:
  otlp_grpc/ezgh:
    endpoint: obs-ingest.ezghcloud.com:443
    headers:
      authorization: "Bearer ${env:EZGH_INGEST_KEY}"
    sending_queue:
      batch:
        sizer: bytes
        min_size: 1048576
        max_size: 4194304
        flush_timeout: 5s

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [otlp_grpc/ezgh]

Several projects from one Collector

An ingest key sends to one project. A Collector that serves several projects has one exporter per project, each with its own key, and routes records to them with the routing connector. This example routes by Kubernetes namespace: payments goes to one project and everything else to another.

connectors:
  routing:
    default_pipelines: [logs/web]
    table:
      - condition: resource.attributes["k8s.namespace.name"] == "payments"
        pipelines: [logs/payments]

exporters:
  otlp_grpc/web:
    endpoint: obs-ingest.ezghcloud.com:443
    headers:
      authorization: "Bearer ${env:EZGH_INGEST_KEY_WEB}"
    sending_queue:
      batch: { sizer: bytes, min_size: 1048576, max_size: 4194304, flush_timeout: 5s }
  otlp_grpc/payments:
    endpoint: obs-ingest.ezghcloud.com:443
    headers:
      authorization: "Bearer ${env:EZGH_INGEST_KEY_PAYMENTS}"
    sending_queue:
      batch: { sizer: bytes, min_size: 1048576, max_size: 4194304, flush_timeout: 5s }

service:
  pipelines:
    logs/in:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [routing]
    logs/web:
      receivers: [routing]
      exporters: [otlp_grpc/web]
    logs/payments:
      receivers: [routing]
      exporters: [otlp_grpc/payments]

Add the otlp receiver and memory_limiter processor from the gateway example. Use otlp_http exporters the same way.

Sending to cold storage

To write a pipeline’s records straight to cold storage (for a dataset that allows it), add the header to its exporter:

    headers:
      authorization: "Bearer ${env:EZGH_INGEST_KEY}"
      x-ezgh-placement: cold