OpenTelemetry Collectors: The Agent + Gateway Pattern

The agent and gateway pattern is a two-tier OpenTelemetry Collector deployment: a small agent collector on every box that receives OTLP from the local app, batches, and forwards, plus one central gateway collector that fans out to the backends (traces to Tempo, logs to Loki, metrics to Prometheus). Apps only ever know a local address; only the gateway knows where telemetry actually lives.

Every OpenTelemetry tutorial ends the same way: one app, one collector, one Grafana. I wrote one of those myself in monitoring .NET applications with OpenTelemetry and Grafana.

The design gets interesting when the system grows past one machine. Katabench runs its API and its grading workers on separate boxes, with the monitoring stack on a third, and that split forces the question the tutorials skip: who sends telemetry where?

The Topology

Each application box runs a small collector next to the app. The app exports OTLP to http://otel-collector:4317, a name that resolves inside the box's own compose network, and knows nothing else.

The agents forward to the gateway on the monitoring box over a private Tailscale network. The gateway is the only component that knows the backends exist: traces go to Tempo, logs to Loki over its native OTLP ingest, and metrics are exposed on a scrape endpoint for Prometheus.

Adding a box to the system costs zero monitoring configuration. Telemetry is push-based, service.instance.id distinguishes instances, and the new box's agent just needs the one gateway address.

Why Run an Agent on Every Box?

The agent config is short enough to show almost in full:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  # Hard ceiling on the collector's own memory. Spikes are dropped, not queued.
  memory_limiter:
    check_interval: 1s
    limit_mib: 96
    spike_limit_mib: 32
  batch:

exporters:
  otlp/gateway:
    endpoint: ${env:OTEL_ENDPOINT}   # no default: missing value fails at boot
    tls:
      insecure: true                 # the link is WireGuard-encrypted already
    retry_on_failure:
      enabled: true
    sending_queue:
      enabled: true
      queue_size: 5000

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/gateway]
    # metrics and logs: same shape

Four deliberate decisions live in those thirty lines:

• The app only knows a local address. No cross-box IP is baked into app config. When the monitoring stack moves, app deployments don't change.
• A sending queue plus retry rides out a gateway or network blip instead of dropping spans. Direct app-to-backend export has no such buffer.
• memory_limiter runs first in every pipeline, and the numbers matter on shared boxes. My worker box's RAM is budgeted for sandbox runs, so the agent gets a hard 96 MB ceiling and sheds load rather than competing with the actual workload for memory. An observability sidecar that can trigger the OOM killer is a self-own.
• ${env:OTEL_ENDPOINT} has no default. A missing value crashes the collector at boot. The alternative is a collector that starts fine and silently swallows telemetry, which you discover three weeks later, mid-incident.

One more property worth copying: the agent publishes no host ports. It is reachable only by the app container on the same compose network, so no box exposes an OTLP ingest surface, not even on the private network.

The Gateway: One Ingest Point, Per-Signal Fan-Out

The gateway is where the per-signal routing happens:

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889        # scrape surface for Prometheus
  otlphttp/loki:
    endpoint: http://loki:3100/otlp # Loki ingests OTLP natively since 3.0
  otlp/tempo:
    endpoint: tempo:4317

service:
  pipelines:
    traces:
      receivers: [otlp, faro]
      processors: [batch]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp, faro]
      processors: [batch]
      exporters: [otlphttp/loki]

That faro receiver is the bonus a central gateway buys you. The browser SPA reports real-user monitoring (JS errors, Web Vitals, fetch spans) through Grafana Faro into the same collector, sharing the traces and logs pipelines. A browser fetch and the API span it triggered land in one Tempo trace, stitched by the propagated traceparent. It is the single observability endpoint that must be public, because it is called from users' browsers; everything else stays private.

Key Takeaways

When do you actually need this? Rules of thumb, from running it:

• One box, one app: a single collector is fine. Don't build the two-tier setup for a monolith on one VPS.
• Two or more boxes, or any box whose RAM you care about: add agents. The local buffering alone pays for the extra container the first time your monitoring box restarts during a deploy and no telemetry is lost.
• Browser RUM, or multiple teams sending telemetry: you want the gateway as the single, controlled ingest point regardless of box count.

The nice thing about the pattern is that migrating to it is invisible to your applications. The .NET side of the wiring never changes: the OTLP exporter still points at whatever OTEL_EXPORTER_OTLP_ENDPOINT says, which is exactly how it worked back when the whole stack was one docker-compose file (the setup from introduction to distributed tracing with OpenTelemetry in .NET). Only the address changed.