# How I Use NATS JetStream as a Job Queue in .NET

> My first job queue was a Redis list, and it had two ways to silently lose a job: a worker crash after LPOP, and a broker restart. NATS JetStream closed both with a work-queue stream, a durable pull consumer, and one ordering rule: publish the result before you ack the job. Here is the whole design, small enough to read in five minutes.

Published: 2026-07-10. Author: Milan Jovanović.

Canonical: https://milanjovanovic.tech/blog/nats-jetstream-job-queue-dotnet

My job queue on **NATS JetStream** is two file-backed streams: a work-queue stream where a pool of workers competes on one durable pull consumer, and a short-retention results stream that the API awaits for the outcome.
Jobs survive worker crashes and broker restarts, and one ordering rule keeps results safe: publish the result before you ack the job.

[**Katabench**](https://katabench.com) grades code submissions in a worker pool: the API accepts a submission, a worker compiles and executes it in a sandbox, and the user waits for the verdict.
Between the API and the workers sits a queue, and the hard constraint I set for it was simple: a user must always get their result or a retriable error, **never a silent loss**.

My first transport failed that constraint twice, and the fix is a nice case study in what a real job queue gives you.
The whole design fits in one article.

## Where Does the Redis List Design Lose Jobs?

Version one was the classic minimal queue: a Redis list.
The API does `RPUSH`, workers poll with `LPOP`, results go into a per-job key with a TTL.

It has two silent loss modes, and both are structural:

- **`LPOP` removes the job with no acknowledgment.** A worker that crashes mid-job (and mine run untrusted code, so crashing is normal operation) takes the job with it. Nothing redelivers it.
- **No persistence.** An in-memory Redis restart or redeploy wipes the queue and every in-flight result.

You can patch around both (use `LMOVE` to a processing list, add AOF persistence, build a sweeper for stuck jobs), but at that point you are hand-building acknowledgments and durability.
That is exactly the feature set of a real message broker.

I picked **NATS JetStream** over RabbitMQ for one operational reason: it is a single lightweight binary that is trivial to run on a VPS, and it plugs straight into the Prometheus and Grafana stack I already had.
I covered the fundamentals in [**getting started with NATS JetStream in .NET**](https://milanjovanovic.tech/blog/getting-started-with-nats-jetstream-in-dotnet); this is what the production shape looks like.

## The Shape: Two Streams

![The two-stream job queue: the API publishes grading jobs to a file-backed work-queue stream, a pool of workers competes on one durable pull consumer, and each worker publishes the outcome to a short-retention results stream that the API awaits filtered by job id](https://milanjovanovic.tech/blogs/articles/nats-jetstream-job-queue-dotnet/job-queue-streams.png)

Everything is built on two file-backed streams:

```csharp
var js = new NatsJSContext(connection);

// The job queue: a message is removed once a worker acks it.
await js.CreateStreamAsync(new StreamConfig("GRADING_JOBS", ["grading.jobs"])
{
    Retention = StreamConfigRetention.Workqueue,
    Storage = StreamConfigStorage.File
});

// Results: short retention, one subject per job id.
await js.CreateStreamAsync(new StreamConfig("GRADING_RESULTS", ["grading.results.>"])
{
    Retention = StreamConfigRetention.Limits,
    MaxAge = TimeSpan.FromMinutes(5),
    Storage = StreamConfigStorage.File
});
```

The **jobs stream** uses work-queue retention: JetStream deletes a message once a consumer acks it, so the stream behaves like a queue instead of a log.
All workers compete on **one shared durable pull consumer**, which is how you get the competing-consumers pattern; adding a worker is just starting another process, with zero configuration.

The **results stream** is the part most people skip.
The worker publishes each outcome to `grading.results.<jobId>`, and the API awaits it with an ephemeral ordered consumer filtered to that one subject.
Because the stream retains messages for a few minutes, a result that was published **before** the API started waiting is still delivered.
That closes an entire class of races that a fire-and-forget reply channel has.

## The One Ordering Rule That Matters

The worker loop looks like this, and the order of the last two lines is the whole reliability story:

```csharp
var consumer = await js.CreateOrUpdateConsumerAsync("GRADING_JOBS",
    new ConsumerConfig("workers")
    {
        AckWait = TimeSpan.FromSeconds(60), // redeliver if no ack in time
        MaxDeliver = 4                      // bounded poison retry
    });

await foreach (var msg in consumer.ConsumeAsync<GradingJob>(cancellationToken: ct))
{
    GradingJobOutcome outcome = await grader.GradeAsync(msg.Data, ct);

    // Publish the result FIRST, then ack the job.
    await js.PublishAsync($"grading.results.{msg.Data.JobId}", outcome, cancellationToken: ct);
    await msg.AckAsync(cancellationToken: ct);
}
```

**Publish the result before you ack the job.**
Walk the failure windows and you'll see why:

- Crash **before** the publish: the job is un-acked, so JetStream redelivers it after `AckWait` and another worker grades it. Nothing is lost.
- Crash **between** publish and ack: the job is redelivered and graded again, and a second identical result is published. Grading is idempotent and keyed by job id, so the duplicate is harmless.

Ack first and you reopen the Redis hole: a crash after the ack but before the publish loses the result with no way to recover it.
At-least-once delivery plus idempotent processing beats exactly-once promises every time.

`MaxDeliver` handles the adversarial case.
A submission that reproducibly crashes its worker (I run untrusted code, so this is a when, not an if) is redelivered a bounded number of times and then dropped, and JetStream emits a `MAX_DELIVERIES` advisory that monitoring alerts on.
Without it, one poison job crash-loops your whole worker pool forever.

## Masking It All Behind a Synchronous API

From the user's perspective nothing here is asynchronous: the API holds the HTTP request open, awaits the result subject up to a dispatch budget, and returns a retriable **503** on timeout.
Every failure mode above degrades to "try again", never to a silently missing result.

Two honest limitations, so you can steal this design with eyes open:

- **A single NATS node is a single point of failure.** Jobs survive a restart (file-backed streams), but not a dead disk. Clustered JetStream with RAFT replicas is the fix when you outgrow one box; at my volume, one box is fine.
- **The results stream is retention-bounded.** If a client should be able to fetch a result hours later, the queue is the wrong home for it; persist results to Postgres and treat the stream purely as transport.

And if you want to see the queue from the user's side, every submission on [**Katabench**](https://katabench.com) rides through it, a few hundred milliseconds at a time.

## Summary

- **Work-queue retention** makes the jobs stream behave like a queue: acking a job deletes it, and each job is processed by exactly one worker.
- **One shared durable pull consumer** gives you competing consumers; adding a worker is just starting another process.
- **Publish the result before you ack the job.** Every crash window then degrades to a redelivery, and idempotent grading keyed by job id makes duplicates harmless.
- **`AckWait` bounds a crashed worker** (the job is redelivered to another worker) and **`MaxDeliver` bounds a poison job** (dropped after a bounded number of attempts, with a `MAX_DELIVERIES` advisory to alert on).
- **The API stays synchronous**: it awaits the results stream and returns a retriable 503 on timeout, never a silent loss.

## Frequently asked questions

### Why use NATS JetStream instead of Redis for a job queue?

A Redis list gives you LPUSH and LPOP but no acknowledgment: once a worker pops a job, a crash loses it, and without AOF persistence a Redis restart wipes the whole queue. JetStream gives you file-backed streams and consumer acknowledgments, so an unacked job is redelivered after a visibility window and jobs survive broker restarts. As a single lightweight binary, NATS is also much easier to operate on a VPS than a full RabbitMQ deployment.

### What happens if a worker crashes in the middle of a job?

The job stays unacknowledged, so JetStream redelivers it to another worker after the AckWait window expires. As long as job processing is idempotent, keyed by a job id, the redelivered job is simply processed again and the client never notices.

### How do you handle poison messages in NATS JetStream?

Set MaxDeliver on the consumer. A job that reproducibly crashes its worker is redelivered a bounded number of times and then dropped, and JetStream emits a MAX_DELIVERIES advisory event you can alert on. The waiting client gets a timeout instead of an infinite crash loop.

### What is work-queue retention in JetStream?

A retention policy where a message is removed from the stream once a consumer acknowledges it. That makes the stream behave like a classic job queue: each job is processed by exactly one member of the worker pool, and acknowledged jobs do not accumulate.