How I Use NATS JetStream as a Job Queue in .NET

My job queue on NATS JetStream is two file-backed streams: a work-queue stream where a pool of workers competes on one durable pull consumer, and a short-retention results stream that the API awaits for the outcome. Jobs survive worker crashes and broker restarts, and one ordering rule keeps results safe: publish the result before you ack the job.

Katabench grades code submissions in a worker pool: the API accepts a submission, a worker compiles and executes it in a sandbox, and the user waits for the verdict. Between the API and the workers sits a queue, and the hard constraint I set for it was simple: a user must always get their result or a retriable error, never a silent loss.

My first transport failed that constraint twice, and the fix is a nice case study in what a real job queue gives you. The whole design fits in one article.

Where Does the Redis List Design Lose Jobs?

Version one was the classic minimal queue: a Redis list. The API does RPUSH, workers poll with LPOP, results go into a per-job key with a TTL.

It has two silent loss modes, and both are structural:

• LPOP removes the job with no acknowledgment. A worker that crashes mid-job (and mine run untrusted code, so crashing is normal operation) takes the job with it. Nothing redelivers it.
• No persistence. An in-memory Redis restart or redeploy wipes the queue and every in-flight result.

You can patch around both (use LMOVE to a processing list, add AOF persistence, build a sweeper for stuck jobs), but at that point you are hand-building acknowledgments and durability. That is exactly the feature set of a real message broker.

I picked NATS JetStream over RabbitMQ for one operational reason: it is a single lightweight binary that is trivial to run on a VPS, and it plugs straight into the Prometheus and Grafana stack I already had. I covered the fundamentals in getting started with NATS JetStream in .NET; this is what the production shape looks like.

The Shape: Two Streams

Everything is built on two file-backed streams:

var js = new NatsJSContext(connection);

// The job queue: a message is removed once a worker acks it.
await js.CreateStreamAsync(new StreamConfig("GRADING_JOBS", ["grading.jobs"])
{
    Retention = StreamConfigRetention.Workqueue,
    Storage = StreamConfigStorage.File
});

// Results: short retention, one subject per job id.
await js.CreateStreamAsync(new StreamConfig("GRADING_RESULTS", ["grading.results.>"])
{
    Retention = StreamConfigRetention.Limits,
    MaxAge = TimeSpan.FromMinutes(5),
    Storage = StreamConfigStorage.File
});

The jobs stream uses work-queue retention: JetStream deletes a message once a consumer acks it, so the stream behaves like a queue instead of a log. All workers compete on one shared durable pull consumer, which is how you get the competing-consumers pattern; adding a worker is just starting another process, with zero configuration.

The results stream is the part most people skip. The worker publishes each outcome to grading.results.<jobId>, and the API awaits it with an ephemeral ordered consumer filtered to that one subject. Because the stream retains messages for a few minutes, a result that was published before the API started waiting is still delivered. That closes an entire class of races that a fire-and-forget reply channel has.

The One Ordering Rule That Matters

The worker loop looks like this, and the order of the last two lines is the whole reliability story:

var consumer = await js.CreateOrUpdateConsumerAsync("GRADING_JOBS",
    new ConsumerConfig("workers")
    {
        AckWait = TimeSpan.FromSeconds(60), // redeliver if no ack in time
        MaxDeliver = 4                      // bounded poison retry
    });

await foreach (var msg in consumer.ConsumeAsync<GradingJob>(cancellationToken: ct))
{
    GradingJobOutcome outcome = await grader.GradeAsync(msg.Data, ct);

    // Publish the result FIRST, then ack the job.
    await js.PublishAsync($"grading.results.{msg.Data.JobId}", outcome, cancellationToken: ct);
    await msg.AckAsync(cancellationToken: ct);
}

Publish the result before you ack the job. Walk the failure windows and you'll see why:

• Crash before the publish: the job is un-acked, so JetStream redelivers it after AckWait and another worker grades it. Nothing is lost.
• Crash between publish and ack: the job is redelivered and graded again, and a second identical result is published. Grading is idempotent and keyed by job id, so the duplicate is harmless.

Ack first and you reopen the Redis hole: a crash after the ack but before the publish loses the result with no way to recover it. At-least-once delivery plus idempotent processing beats exactly-once promises every time.

MaxDeliver handles the adversarial case. A submission that reproducibly crashes its worker (I run untrusted code, so this is a when, not an if) is redelivered a bounded number of times and then dropped, and JetStream emits a MAX_DELIVERIES advisory that monitoring alerts on. Without it, one poison job crash-loops your whole worker pool forever.

Masking It All Behind a Synchronous API

From the user's perspective nothing here is asynchronous: the API holds the HTTP request open, awaits the result subject up to a dispatch budget, and returns a retriable 503 on timeout. Every failure mode above degrades to "try again", never to a silently missing result.

Two honest limitations, so you can steal this design with eyes open:

• A single NATS node is a single point of failure. Jobs survive a restart (file-backed streams), but not a dead disk. Clustered JetStream with RAFT replicas is the fix when you outgrow one box; at my volume, one box is fine.
• The results stream is retention-bounded. If a client should be able to fetch a result hours later, the queue is the wrong home for it; persist results to Postgres and treat the stream purely as transport.

And if you want to see the queue from the user's side, every submission on Katabench rides through it, a few hundred milliseconds at a time.

Summary

• Work-queue retention makes the jobs stream behave like a queue: acking a job deletes it, and each job is processed by exactly one worker.
• One shared durable pull consumer gives you competing consumers; adding a worker is just starting another process.
• Publish the result before you ack the job. Every crash window then degrades to a redelivery, and idempotent grading keyed by job id makes duplicates harmless.
• AckWait bounds a crashed worker (the job is redelivered to another worker) and MaxDeliver bounds a poison job (dropped after a bounded number of attempts, with a MAX_DELIVERIES advisory to alert on).
• The API stays synchronous: it awaits the results stream and returns a retriable 503 on timeout, never a silent loss.