Zstandard Compression in .NET 11: Read and Write .zst Files

.NET 11 adds ZstandardStream to System.IO.Compression. It writes standard .zst files that the zstd command-line tool can read, and it can decode files that tool produces. With the same encoder settings, the 14-byte CSV below grows to 27 bytes, while a 10,000-row export shrinks to less than a tenth of its original size.

How Do You Compress and Decompress a File?

To send a single orders.csv file to an importer, write its bytes into a Zstandard frame. If you need to preserve several filenames and directories, put them in a tar archive before compression. That's what a .tar.zst file contains.

Create a console app with the .NET 11 SDK.

dotnet new console -n ZstdExport -f net11.0
cd ZstdExport

In Program.cs, give both pack and unpack an input path and a new output path. FileMode.CreateNew prevents a typo in the output name from overwriting an existing export.

using System.IO.Compression;

if (args.Length != 3 || args[0] is not ("pack" or "unpack"))
{
    Console.Error.WriteLine("Usage: ZstdExport <pack|unpack> <input> <new-output>");
    return 1;
}

using var input = File.OpenRead(args[1]);
using var output = new FileStream(args[2], FileMode.CreateNew, FileAccess.Write);

if (args[0] == "pack")
{
    var options = new ZstandardCompressionOptions { AppendChecksum = true };
    using var encoder = new ZstandardStream(output, options, leaveOpen: true);
    input.CopyTo(encoder);
}
else
{
    using var decoder = new ZstandardStream(input, CompressionMode.Decompress, leaveOpen: true);
    decoder.CopyTo(output);
}

return 0;

pack copies the input into the encoder. unpack copies data from the decoder into the destination file. Neither path needs to load the whole file into memory.

At the closing brace, disposing the encoder writes the end of the frame. If another job watches the export directory, write to a temporary path there and rename the file after pack exits successfully. Otherwise, the importer can pick up a file while its last block is still buffered.

Verify the File With a Second Reader

In a Unix shell, make a one-row CSV, run both commands, and compare the original bytes with the restored file. A successful cmp prints nothing.

printf 'id,total\n1,42\n' > orders.csv
dotnet run -- pack orders.csv orders.csv.zst
dotnet run -- unpack orders.csv.zst restored.csv
cmp orders.csv restored.csv

Now open the .NET output with the Zstandard CLI. Then switch writers: compress the CSV with zstd and feed that file to unpack.

zstd -dc orders.csv.zst > from-dotnet.csv
cmp orders.csv from-dotnet.csv
zstd -q orders.csv -o from-cli.zst
dotnet run -- unpack from-cli.zst from-cli.csv
cmp orders.csv from-cli.csv

The example sets AppendChecksum so a reader can detect accidental corruption. Flipping the last bit of the checksum in this file makes unpack throw InvalidDataException. If decompression fails partway through a large import, discard the partial output before retrying.

When Is Compression Worth the Work?

The one-row CSV above takes 14 bytes as plain text and 27 bytes as a .zst file with the .NET 11.0.0 encoder and checksum enabled. For comparison, generate 10,000 rows with the same two columns.

awk 'BEGIN {print "id,total"; for (i=1; i<=10000; i++) print i ",42"}' > batch.csv
dotnet run -- pack batch.csv batch.csv.zst
wc -c batch.csv batch.csv.zst

The 10,000-row file took 78,903 bytes before compression and 6,892 bytes afterward with the same settings. The repeated ,42 gives the encoder something to reuse across rows. Compressing each row separately would add a new frame header and checksum every time.

Keep the default quality for the first run on a real export. If the upload dominates the job, try a higher ZstandardCompressionOptions.Quality and compare the total time until the upload finishes. Higher settings spend more CPU looking for a smaller encoding, and the highest settings also require more memory. Microsoft documents the available settings in ZstandardCompressionOptions.

Can Every Client Read the Export?

Check the importer's supported formats before switching a shared export from .gz to .zst. A reader built around GZipStream needs a Zstandard decoder to open the new file. The example uses no external dictionary. If you later train one for small records, distribute the matching dictionary to every reader and retain it for as long as those records need to be readable.

Microsoft marks ZstandardStream unsupported on browser and WASI targets. If a Blazor WebAssembly client opens downloaded files itself, choose a decoder supported by that client or perform decompression on the server.

If the importer asks an API for CSV rather than downloading a stored file, ASP.NET Core response compression lets it advertise the encodings it can read. The Zstandard upload and download example sends the downloaded bytes back in an upload request.

Summary

Before switching a nightly export to .zst, send one full-sized file through the partner's importer and compare the decoded bytes with the original. If you're tuning compression quality, time the whole job through the upload. A smaller file can still arrive later if the encoder spends longer compressing it than you save on the transfer.