Clock Skew: Why You Can't Trust Timestamps Across Machines

Clock skew is the difference between machines' clocks at the same instant. It can make timestamp sorting disagree with event order even when both services are healthy. Use serialized log positions, logical clocks, or record versions when ordering affects correctness.

For example, suppose the following events occur: Service A stamps an event with its clock. Service B stamps a later event with its clock. Your code sorts by timestamp, and B's event comes out first, because B's clock runs 40 milliseconds behind.

Even with time synchronization, timestamps cannot establish an event order when their uncertainty intervals overlap.

How Much Can Clocks Disagree?

The accuracy you can assume depends on your deployment:

• Servers using NTP: accuracy depends on the clock source and network path. Synchronization alone is not a documented bound for your fleet.
• Cloud VMs: host pauses, migrations, and provider-specific time sources affect clock behavior. Check the time service's guarantees for your instance type.
• Failure cases: an unreachable NTP server, a misconfigured container host, or a VM restored from a snapshot can leave clocks far apart.
• Steps and slews: when NTP corrects a drifted clock, it either slews (speeds/slows the clock gradually) or steps it, a discontinuous jump, possibly backward. Wall-clock time on one machine is not even monotonic.

Different leap-second smear policies can also produce different times throughout a smear window.

A timestamp is a measurement with uncertainty. Two events 5 ms apart cannot be ordered reliably if the combined clock uncertainty is larger than that interval.

Where This Bites Real Systems

Last-write-wins conflict resolution. Two services update the same record; the "later" timestamp wins. With 20 ms of skew, a write that happened first can carry a later timestamp and overwrite the actual latest data. This is a real data-loss mode in timestamp-based conflict resolution, and the overwrite produces no error or log entry.

Event ordering in projections. An event stream consumer sorts by OccurredAt to rebuild state. OrderCancelled (stamped by host A) sorts before OrderPlaced (stamped by host B), and your projection either throws or, worse, builds the wrong state without an error. I walked through the broader ordering problem in solving message ordering from first principles; clock skew is one of the reasons "just sort by time" is not on the solution list.

Cache and token expiry at boundaries. A token issued by one server is validated by another whose clock runs early, and a just-issued token is "expired". Microsoft IdentityModel's default ClockSkew is five minutes. Set tolerance according to your issuer and security requirements; increasing it also extends acceptance around token lifetime boundaries.

Distributed locks and leases. "The lease is valid until 12:00:00.500" means different things on machines that disagree on what 12:00:00.500 is. Lease-based coordination must build in skew margins, which is part of why distributed locking is harder than it looks.

Measuring duration with wall clocks. This bug needs only one machine, because DateTime.UtcNow can step backward mid-measurement.

// Wrong: wall clock, can step backward or jump forward
var wallStart = DateTime.UtcNow;
await Task.Delay(100);
var wallElapsed = DateTime.UtcNow - wallStart;

// Right: monotonic clock, immune to NTP corrections
var start = System.Diagnostics.Stopwatch.GetTimestamp();
await Task.Delay(100);
var elapsed = System.Diagnostics.Stopwatch.GetElapsedTime(start);

Stopwatch (and Environment.TickCount64) read a monotonic clock that only moves forward. Use wall-clock time for timestamps and Stopwatch for elapsed durations within a process.

Ordering Without Trusting Clocks

Choose an ordering guarantee that matches the operation: a log position, a causal relationship, or a version on a record.

Single-writer sequencing. Let one serialized writer assign positions when you need an authoritative sequence. An identity column is useful for generating positions, but concurrent database transactions require extra care:

CREATE TABLE outbox_messages (
    position   bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    payload    jsonb NOT NULL,
    created_at timestamptz NOT NULL DEFAULT now() -- for humans, not for ordering
);

PostgreSQL sequences allocate numbers outside transaction rollback semantics. Transaction A can allocate position 10, transaction B can allocate and commit 11, and A can commit later. A poller that advances its cursor to 11 can permanently miss 10.

A local PostgreSQL 18 check held the lower-ID insert open while committing a second connection's insert:

PostgreSQL identity: higher ID committed while lower ID was uncommitted, confirming cursor-skip risk.

Serialize appends through commit, use a committed log position such as a broker offset, or poll unpublished outbox rows without assuming identity values are commit order. Kafka gives each partition an append order, but producers still need to send causally related events in the required order.

Logical clocks (Lamport). When there is no single writer, you can still capture causality: every node keeps a counter, increments it on each event, attaches it to messages, and on receipt sets counter = max(local, received) + 1.

public sealed class LamportClock
{
    private long _time;

    public long Tick() => Interlocked.Increment(ref _time);

    public long Observe(long received)
    {
        long current, next;
        do
        {
            current = Interlocked.Read(ref _time);
            next = Math.Max(current, received) + 1;
        }
        while (Interlocked.CompareExchange(ref _time, next, current) != current);

        return next;
    }
}

If event X caused event Y, X's Lamport timestamp is guaranteed to be smaller than Y's, with no clock sync required. (The converse does not hold, and vector clocks exist for when you need to detect concurrency, at the cost of carrying a counter per node.)

Version numbers per record. For concurrent updates, use an integer version on the row, incremented on every write, with writes conditional on the version they read. Optimistic concurrency control detects a stale update; your application must then reject, merge, or retry it.

What About GUID v7 and Time-Ordered IDs?

Guid.CreateVersion7(), available since .NET 9, embeds a Unix millisecond timestamp. It can improve index locality when the database's GUID comparison order matches the UUID layout. I covered it in GUID v7 in .NET.

But the IDs are only roughly sortable. IDs minted on different machines inherit those machines' skew, and within one millisecond ordering is random. Use v7 GUIDs for index-friendly identifiers, not as a distributed ordering primitive. The same verdict applies to any "timestamp + random bits" scheme (ULID, Snowflake without coordinated clocks).

Practical Rules for .NET Systems

• Stamp everything in UTC, and prefer DateTimeOffset over DateTime so the offset is explicit; see DateTime vs DateTimeOffset.
• Durations: monotonic only. Stopwatch.GetElapsedTime, Environment.TickCount64, or TimeProvider.GetTimestamp().
• Inject time. Route time access through .NET 8's TimeProvider instead of static DateTime.UtcNow calls. Beyond testability (covered in TimeProvider in .NET), it gives you a single place to apply time policy.
• Define time assumptions at trust boundaries. Monitor offset and synchronization failures. Token acceptance may need a tolerance; lease holders need conservative deadlines, because extending a lease locally would make stale leadership last longer.
• Avoid cross-machine timestamps as the sole correctness check. For deciding which write wins or how events are ordered, prefer versions, serialized log positions, or logical clocks. Time-based protocols need explicit error bounds and failure handling.

Summary

• Synchronized clocks can still disagree, and wall-clock corrections can move time backward.
• Do not infer an exact cross-machine order from timestamps whose uncertainty overlaps.
• Get ordering from serialized writers or log offsets, causal ordering from logical clocks, and conflict detection from version numbers.
• In .NET specifically: DateTimeOffset in UTC for "when", Stopwatch/TimeProvider.GetTimestamp for "how long", and TimeProvider as the seam that keeps time testable.

Test with skewed and backward-moving wall clocks. Your ordering and concurrency tests should still pass without depending on the local machine's current time.