Message Ordering With Partition Keys and Sequences in .NET

Reliable message ordering requires agreement between the producer, dispatcher, broker, and consumer. Use a stable entity key to group related messages and an authoritative per-entity sequence to detect gaps or duplicates. Process that key serially when every event matters; complete snapshot events can instead use atomic version checks to reject stale updates.

Why Do Messages Arrive Out of Order?

A broker can deliver messages in order while application handlers finish them out of order. Two workers may receive events 1 and 2, but worker 2 commits its database update first. Retries and multiple publishers introduce additional ordering changes.

The competing consumers pattern increases throughput, so its concurrency has to match the business requirement. Order 123 and order 456 can usually progress independently. Events within order 123 may need to be applied in sequence.

A global order is possible in some systems, but establishing and consuming it adds coordination and constrains parallelism. Start by identifying the smallest stream whose effects depend on previous events. Solving message ordering from first principles develops that reasoning in more detail.

Partition by the Required Stream

A partition key routes related events into the same ordered stream. The handler must also avoid concurrent processing within that stream if completion order matters. Neither routing nor a broker delivery guarantee alone ensures database commit order.

The broker mechanisms differ:

• Kafka orders records within a partition. A stable key normally selects that partition; partition-count changes and asynchronous handler concurrency need care.
• Azure Service Bus sessions group related messages by SessionId and give a receiver an exclusive session lock. Configure the session-enabled queue or subscription and process one message at a time per session when effects must remain ordered.
• RabbitMQ queues preserve enqueue order under documented conditions. Single active consumer limits which consumer receives deliveries, but your handler still needs serial execution. Sharding keys into separate queues provides parallel streams.
• Amazon SQS FIFO queues group messages by MessageGroupId. Process messages within a received group in order, and account for visibility expiry and redelivery.

These mechanisms cannot repair producer reordering. If two outbox dispatchers send version 2 before version 1, an ordered broker can faithfully preserve that wrong order. Partition dispatch ownership by stream or let consumers detect and resolve gaps.

MassTransit's UsePartitioner middleware coordinates handlers inside a single process. It does not establish exclusive ownership across application instances consuming the same queue. Use broker routing or durable coordination when scaling that consumer out.

Assign Sequences and Store Events Together

A source-generated sequence lets the consumer distinguish a duplicate from a missing earlier event. It must be assigned atomically with the event record. Otherwise a crash after incrementing a counter can leave a permanent gap.

The following .NET 10 example uses Npgsql and PostgreSQL to assign a sequence and store an outbox event in one transaction:

dotnet new console -n SequencedOutboxDemo -f net10.0
cd SequencedOutboxDemo
dotnet add package Npgsql --version 10.0.0
docker run --rm --name sequence-postgres -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres:18

After PostgreSQL is ready, put this in Program.cs and run dotnet run:

using Npgsql;

await using var dataSource = NpgsqlDataSource.Create(
    "Host=localhost;Database=postgres;Username=postgres;Password=postgres");

await using (var schema = dataSource.CreateCommand("""
    CREATE TABLE IF NOT EXISTS event_sequences (
        entity_id UUID PRIMARY KEY,
        last_sequence BIGINT NOT NULL
    );
    CREATE TABLE IF NOT EXISTS sequenced_outbox (
        id UUID PRIMARY KEY,
        entity_id UUID NOT NULL,
        sequence_number BIGINT NOT NULL,
        payload TEXT NOT NULL,
        processed_at TIMESTAMPTZ NULL,
        UNIQUE (entity_id, sequence_number)
    );
    """))
{
    await schema.ExecuteNonQueryAsync();
}

var orderId = Guid.NewGuid();
Console.WriteLine(await AppendAsync(orderId, "OrderCreated"));
Console.WriteLine(await AppendAsync(orderId, "OrderConfirmed"));

async Task<long> AppendAsync(Guid entityId, string payload)
{
    await using var connection = await dataSource.OpenConnectionAsync();
    await using var transaction = await connection.BeginTransactionAsync();

    await using var next = new NpgsqlCommand("""
        INSERT INTO event_sequences (entity_id, last_sequence)
        VALUES ($1, 1)
        ON CONFLICT (entity_id) DO UPDATE
        SET last_sequence = event_sequences.last_sequence + 1
        RETURNING last_sequence
        """, connection, transaction);
    next.Parameters.AddWithValue(entityId);
    var sequence = (long)(await next.ExecuteScalarAsync())!;

    await using var append = new NpgsqlCommand("""
        INSERT INTO sequenced_outbox (id, entity_id, sequence_number, payload)
        VALUES ($1, $2, $3, $4)
        """, connection, transaction);
    append.Parameters.AddWithValue(Guid.NewGuid());
    append.Parameters.AddWithValue(entityId);
    append.Parameters.AddWithValue(sequence);
    append.Parameters.AddWithValue(payload);
    await append.ExecuteNonQueryAsync();

    await transaction.CommitAsync();
    return sequence;
}

The output is:

1
2

The counter row serializes concurrent writers for that entity. If the append fails, disposing the transaction rolls back the counter increment too. This differs from PostgreSQL sequence objects, whose allocated values are not rolled back and therefore do not provide gapless numbering.

This sample stores the sequence and event; it does not dispatch messages or implement order business rules. When a command also changes business state, perform that change on the same connection and transaction. Use a stable command or request ID to prevent a retried request from appending the same logical event again under a new sequence.

The command executes INSERT ... RETURNING directly. Do not pass it to an EF Core query and compose SingleAsync over it: EF may wrap the SQL in a subquery, which is not valid for this statement shape. The EF Core SQL query documentation explains the composability requirement.

Detect Gaps Without Losing Messages

Suppose a consumer expects sequence 3 but receives sequence 4. It cannot apply 4 yet if its effect depends on 3. It needs a durable policy for retaining or redelivering the early event.

One design stores incoming events in a database inbox and processes them separately:

1. Insert the received event into a pending-event table with a unique (stream_id, sequence) key, then commit before acknowledging broker delivery.
2. In a transaction, lock the stream's cursor row and read the next expected event.
3. Apply that event's database changes and advance the cursor in the same transaction.
4. Continue through contiguous events in bounded batches. If the next sequence is absent, leave later events pending.

Store conflicting payloads for the same stream and sequence as an error instead of overwriting them. Set limits on backlog size, alert on old gaps, and define replay or reconciliation procedures. An event dead-lettered by another stage can otherwise block this stream indefinitely.

Both the cursor and pending payloads must survive a restart. An in-memory sorted list loses acknowledged events when the process stops, even if you persist the cursor separately. A ConcurrentDictionary also does not make a mutable list inside it or a multi-step handler atomic.

This design only works when the consumer is meant to receive every sequence in that stream. If its subscription filters out some event types, missing numbers may be expected. Use a sequence scoped to the consumer's stream or a protocol that distinguishes filtered events from lost ones.

When Can a Consumer Ignore Order?

Complete snapshots can use an atomic "apply only if newer" write. The eventual consistency example demonstrates a PostgreSQL upsert that prevents an older version overwriting a newer projection, including concurrent first inserts. Every snapshot must contain all fields that projection needs.

Commutative operations can sometimes be applied in either order. They still need deduplication when repeating an operation changes the result. For example, applying two independent increments can commute, while applying the same increment twice is still incorrect.

Idempotency and ordering solve different problems. An idempotent consumer avoids repeated effects for a duplicate message; it does not make two different messages interchangeable. Do not discard intermediate events from a ledger or a delta-based projection just because a newer sequence has arrived.

Cross-Service Order Is a Workflow Requirement

"Reserve inventory before charging payment" defines a workflow dependency. An entity partition key cannot establish that dependency across unrelated producers and queues. A saga can record that inventory succeeded before sending the payment command.

A state machine does not automatically hold every unexpected message until a matching state appears. In MassTransit, an event not handled by the current state normally faults unless you configure another behavior. Missing instances, duplicate results, and early events need explicit handling and a bounded retry, deferral, or recovery policy.

Persist the workflow state and outgoing command together through an outbox. Otherwise the coordinator can remember that it advanced without having sent the command that makes the next step happen.

Summary

Specify which stream needs ordering and whether every event contributes a required effect. Assign its sequence with the event record, preserve that order through dispatch and handling, and make gaps recoverable.

Use atomic version checks for complete snapshots and deduplication for repeated effects. When events drive a multi-service workflow, record the dependencies and failure paths explicitly instead of relying on queue arrival order.