# OpenTelemetry Collectors: The Agent + Gateway Pattern

> One OpenTelemetry Collector is easy. The interesting design shows up the moment your apps run on more than one machine: a small agent collector on every box, forwarding to one gateway collector that fans out to Tempo, Loki, and Prometheus. Here is the two-tier setup I run in production, including the two config lines that stop the collector from OOM-killing the box it shares with real workloads.

Published: 2026-07-10. Author: Milan Jovanović.

Canonical: https://milanjovanovic.tech/blog/opentelemetry-collector-agent-gateway

The agent and gateway pattern is a two-tier OpenTelemetry Collector deployment: a small **agent** collector on every box that receives OTLP from the local app, batches, and forwards, plus one central **gateway** collector that fans out to the backends (traces to Tempo, logs to Loki, metrics to Prometheus).
Apps only ever know a local address; only the gateway knows where telemetry actually lives.

Every OpenTelemetry tutorial ends the same way: one app, one collector, one Grafana.
I wrote one of those myself in [**monitoring .NET applications with OpenTelemetry and Grafana**](https://milanjovanovic.tech/blog/monitoring-dotnet-applications-with-opentelemetry-and-grafana).

The design gets interesting when the system grows past one machine.
[**Katabench**](https://katabench.com) runs its API and its grading workers on separate boxes, with the monitoring stack on a third, and that split forces the question the tutorials skip: who sends telemetry where?

## The Topology

![The agent and gateway collector topology: the API box and the worker box each run a local agent collector that their app exports OTLP to; both agents forward over the private network to the gateway collector on the monitoring box, which fans traces to Tempo, logs to Loki, and metrics to a Prometheus scrape endpoint, all rendered by Grafana](https://milanjovanovic.tech/blogs/articles/opentelemetry-collector-agent-gateway/agent-gateway-topology.png)

Each application box runs a small collector next to the app.
The app exports OTLP to `http://otel-collector:4317`, a name that resolves inside the box's own compose network, and knows nothing else.

The agents forward to the gateway on the monitoring box over a private [**Tailscale network**](https://milanjovanovic.tech/blog/build-your-own-vpn-with-tailscale).
The gateway is the only component that knows the backends exist: traces go to Tempo, logs to Loki over its native OTLP ingest, and metrics are exposed on a scrape endpoint for Prometheus.

Adding a box to the system costs zero monitoring configuration.
Telemetry is push-based, `service.instance.id` distinguishes instances, and the new box's agent just needs the one gateway address.

## Why Run an Agent on Every Box?

The agent config is short enough to show almost in full:

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  # Hard ceiling on the collector's own memory. Spikes are dropped, not queued.
  memory_limiter:
    check_interval: 1s
    limit_mib: 96
    spike_limit_mib: 32
  batch:

exporters:
  otlp/gateway:
    endpoint: ${env:OTEL_ENDPOINT}   # no default: missing value fails at boot
    tls:
      insecure: true                 # the link is WireGuard-encrypted already
    retry_on_failure:
      enabled: true
    sending_queue:
      enabled: true
      queue_size: 5000

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/gateway]
    # metrics and logs: same shape
```

Four deliberate decisions live in those thirty lines:

- **The app only knows a local address.** No cross-box IP is baked into app config. When the monitoring stack moves, app deployments don't change.
- **A sending queue plus retry rides out a gateway or network blip** instead of dropping spans. Direct app-to-backend export has no such buffer.
- **`memory_limiter` runs first in every pipeline**, and the numbers matter on shared boxes. My worker box's RAM is budgeted for sandbox runs, so the agent gets a hard 96 MB ceiling and sheds load rather than competing with the actual workload for memory. An observability sidecar that can trigger the OOM killer is a self-own.
- **`${env:OTEL_ENDPOINT}` has no default.** A missing value crashes the collector at boot. The alternative is a collector that starts fine and silently swallows telemetry, which you discover three weeks later, mid-incident.

One more property worth copying: the agent publishes **no host ports**.
It is reachable only by the app container on the same compose network, so no box exposes an OTLP ingest surface, not even on the private network.

## The Gateway: One Ingest Point, Per-Signal Fan-Out

The gateway is where the per-signal routing happens:

```yaml
exporters:
  prometheus:
    endpoint: 0.0.0.0:8889        # scrape surface for Prometheus
  otlphttp/loki:
    endpoint: http://loki:3100/otlp # Loki ingests OTLP natively since 3.0
  otlp/tempo:
    endpoint: tempo:4317

service:
  pipelines:
    traces:
      receivers: [otlp, faro]
      processors: [batch]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp, faro]
      processors: [batch]
      exporters: [otlphttp/loki]
```

That `faro` receiver is the bonus a central gateway buys you.
The browser SPA reports real-user monitoring (JS errors, Web Vitals, fetch spans) through Grafana Faro into the **same** collector, sharing the traces and logs pipelines.
A browser fetch and the API span it triggered land in one Tempo trace, stitched by the propagated `traceparent`.
It is the single observability endpoint that must be public, because it is called from users' browsers; everything else stays private.

## Key Takeaways

When do you actually need this?
Rules of thumb, from running it:

- **One box, one app:** a single collector is fine. Don't build the two-tier setup for a monolith on one VPS.
- **Two or more boxes, or any box whose RAM you care about:** add agents. The local buffering alone pays for the extra container the first time your monitoring box restarts during a deploy and no telemetry is lost.
- **Browser RUM, or multiple teams sending telemetry:** you want the gateway as the single, controlled ingest point regardless of box count.

The nice thing about the pattern is that migrating to it is invisible to your applications.
The .NET side of the wiring never changes: the OTLP exporter still points at whatever `OTEL_EXPORTER_OTLP_ENDPOINT` says, which is exactly how it worked back when the whole stack was one docker-compose file (the setup from [**introduction to distributed tracing with OpenTelemetry in .NET**](https://milanjovanovic.tech/blog/introduction-to-distributed-tracing-with-opentelemetry-in-dotnet)).
Only the address changed.

## Frequently asked questions

### What is the difference between an agent and a gateway OpenTelemetry Collector?

An agent runs on the same machine as the application and is a local hop: it receives OTLP from the local app, batches and compresses, and forwards everything to the gateway. The gateway is the central collector that owns the fan-out to the storage backends: traces to Tempo, logs to Loki, metrics exposed for Prometheus. Apps only ever know a local address; only the gateway knows where telemetry actually lives.

### Why not export OTLP directly from the app to the backend?

Direct export couples every app instance to backend addresses, loses telemetry during any backend or network blip, and gives you no local buffering or batching. An agent collector absorbs those failures with a sending queue and retry, ships in compressed batches, and means the app config never changes when the monitoring stack moves.

### What does the memory_limiter processor do?

It enforces a hard ceiling on the collector's own memory usage, dropping telemetry spikes instead of queueing them. It must be the first processor in every pipeline so it can shed load before batching allocates more memory. On a box whose RAM is budgeted for real workloads, it guarantees the collector is never the process that triggers the OOM killer.

### Does the OpenTelemetry Collector need TLS on a private network?

If the link between agent and gateway is already encrypted at a lower layer, for example a WireGuard-based private network like Tailscale, plaintext OTLP with tls.insecure is a reasonable choice. The traffic is encrypted on the wire by WireGuard, and you skip managing certificates for an internal hop.