
<Sponsorships>

[**Boost your .NET development**](https://abp.io/?utm_source=newsletter&utm_medium=affiliate&utm_campaign=milanjovanovic_jan18) with ABP.IO's framework!
Open-source, module ready and built for rapid web app creation.
From microservices to multi-tenancy, ABP covers your needs.
Focus on your business and [**enjoy coding now**](https://abp.io/?utm_source=newsletter&utm_medium=affiliate&utm_campaign=milanjovanovic_jan18)!

Tired of switching contexts while building apps?
[**Hot Design™**](https://platform.uno/docs/articles/studio/Hot%20Design/hot-design-getstarted-guide.html?utm_source=milan-newsletter&utm_medium=email&utm_campaign=HotDesign-A)
combines runtime UI design, live data integration, and cross-platform development to keep you in the flow.
Compatible with Visual Studio, VS Code, and Rider, it simplifies .NET UI development, accelerates your workflow,
and enhances your developer experience—letting you deliver polished apps with ease.
[**Try Hot Design™**](https://platform.uno/waitlist/?utm_source=milan-newsletter&utm_medium=email&utm_campaign=HotDesign-A)
today and experience effortless productivity!

</Sponsorships>

Monoliths get a bad rap in our industry.
We're told they're legacy, that they don't scale, and that we need microservices to succeed.
After spending many years scaling systems from startups to enterprises, I can tell you this isn't true.
A well-designed monolith is often the right architecture choice, and it can scale remarkably well with the right approach.

In my experience building and scaling monolithic systems, I've found that the key to success isn't following trends.
It's understanding your scaling needs and applying the right solutions at the right time.
In this article, I'll share what I've learned about scaling monoliths effectively and when to use each approach.

## Understanding Scale

A monolith puts all your code in one deployable unit.
This brings significant advantages: faster development cycles, simpler debugging, and straightforward deployments.
But as your system grows, you'll face scaling challenges.

<div className="centered">
  <Image
    src="blogs/mnw_125/monolith_system.png"
    width={131}
    height={430}
    alt="Simple monolith system."
  />
</div>

Your database queries slow down as data volume grows. API endpoints that worked
fine with hundreds of users start timing out with thousands. Build times creep
up as your codebase expands. These are natural growing pains that every
successful system faces.

## Vertical Scaling

Vertical scaling means giving your application more resources on a single machine.
It's the simplest scaling strategy and often the most effective first step.
Before diving into complex distributed systems, consider whether upgrading your existing infrastructure could solve your performance problems.

<div className="centered">
  <Image
    src="blogs/mnw_125/vertical_scaling.png"
    width={643}
    height={542}
    alt="Example of vertically scaling a monolith."
  />
</div>

Vertical scaling works particularly well when there are clear resource bottlenecks.
If your CPU is consistently above 80% utilization, adding more cores will help.
If your database is I/O bound, upgrading to faster storage can dramatically improve performance.
Modern cloud platforms make this especially straightforward.
You can often upgrade with a few clicks and minimal downtime.

The benefits of vertical scaling extend beyond just performance.
It maintains your system's simplicity.
You don't need to redesign your architecture, implement new deployment patterns, or manage distributed system complexity.
Your monitoring, debugging, and operational procedures all stay the same.

However, vertical scaling does have limits.
You'll eventually hit a ceiling on what a single machine can handle.
Cloud providers have maximum instance sizes, and costs typically increase exponentially with larger instances.
More importantly, vertical scaling doesn't provide redundancy - you're still running on a single machine that represents a potential single point of failure.

Knowing when to move beyond vertical scaling is crucial.
Watch for these indicators:

- Costs are growing faster than your user base
- You need better redundancy and fault tolerance
- Your deployment downtime is impacting business operations
- Your largest available instance size is approaching 70% utilization

## Horizontal Scaling

[**Horizontal scaling**](horizontally-scaling-aspnetcore-apis-with-yarp-load-balancing) runs multiple instances of your application behind a load balancer.
It's the next step when vertical scaling reaches its limits.
It offers improved fault tolerance and nearly linear scaling capabilities.

<div className="centered">
  <Image
    src="blogs/mnw_125/horizontal_scaling.png"
    width={630}
    height={648}
    alt="Example of horizontally scaling a monolith."
  />
</div>

The key to successful horizontal scaling lies in application design.
Your application must be stateless - each request should contain all the information needed to process it.

This means:

- Authentication should use tokens (like JWTs) rather than server-side sessions
- Cached data should live in a distributed cache

Load balancers are crucial in this architecture.
Whether you call it an [**API Gateway**](implementing-an-api-gateway-for-microservices-with-yarp), Reverse Proxy, or Load Balancer, its job is to distribute traffic across your application instances.
Popular choices include:

- [nginx](https://nginx.org/en/): Powerful, open-source, great for custom configurations
- [YARP](https://microsoft.github.io/reverse-proxy/): Microsoft's .NET reverse proxy, great for .NET applications
- Cloud:
  [AWS ALB](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html),
  [Azure Application Gateway](https://learn.microsoft.com/en-us/azure/application-gateway/overview),
  [Google Cloud Load Balancing](https://cloud.google.com/load-balancing?hl=en)

Horizontal scaling provides several key benefits:

- Better fault tolerance through redundancy
- Ability to handle more concurrent users
- Rolling deployments with zero downtime
- Cost-effective scaling (scale down when traffic is low)

The main challenge in horizontal scaling isn't technical—it's architectural.
Your application needs to be designed for horizontal scaling from the start.
Converting a stateful application to a stateless one often requires significant refactoring.

## Database Scaling

Database scaling is where most monoliths first hit real limitations.
Let's explore each scaling strategy in detail.

### Read Replicas

Read replicas are often your first step in database scaling but come with significant trade-offs.
Read replicas maintain a copy of your primary database that serves read-only traffic.
When you run a query against a replica, you're not competing with writes on your primary database.

Each replica maintains an up-to-date copy of your data through replication.
Changes flow one-way: from primary to replicas.
This means any data written to your primary will eventually show up in your replicas.
That "eventually" is important - you're trading consistency for better read performance.

Most cloud providers make read replicas easy to set up.
AWS RDS and Azure SQL all support read replicas with minimal configuration.
They handle replication, monitoring, and failover for you.

<Image
  src="blogs/mnw_125/database_read_replicas.png"
  width={1854}
  height={1417}
  alt="Example of database read replication for scaling a monolith."
/>

When implementing read replicas, consider:

- Replication lag affects data freshness
- Write volume impacts replication speed
- Geographic location affects latency
- Each replica adds to your costs

### Materialized Views

Sometimes read replicas aren't enough.
Perhaps you need to reshape your data for specific use cases, or you're running complex analytical queries that are slow even on a replica.
This is where materialized views come in.

A materialized view is a pre-computed dataset stored as a table.
Unlike regular views that compute their results on each query, materialized views store their results.
This makes them much faster to query but introduces a new challenge: keeping them up to date.

Materialized views excel at:

- Complex analytical queries
- Data that updates on a schedule
- Aggregations and summaries
- Denormalized data for specific views

The key trade-off is freshness versus performance.
You need to decide how often to refresh your materialized views.
Too often, and you're putting load on your database.
Too rarely, and your data gets stale.

### Database Sharding

Sharding becomes necessary when your database grows beyond what a single instance can handle.
Sharding splits your data across multiple database instances, with each shard containing a distinct subset of your data.
The key to successful sharding lies in choosing the right sharding strategy for your use case.

**Range-based sharding** splits data based on ranges of a key value - for example, customers A-M go to Shard 1, N-Z to Shard 2.
This approach works well with data that has a natural range distribution, like dates or alphabetical order,
but can lead to hotspots if certain ranges see more activity than others.

**Hash-based sharding** applies a hash function to your sharding key to determine which shard holds the data.
The choice of hashing function is crucial.
It must distribute data evenly across your shards to prevent any single shard from becoming a bottleneck.
While this approach provides better data distribution, it makes range-based queries more complex since related data might live on different shards.

**Tenant-based sharding** gives each tenant their own database.
This approach provides natural isolation and makes tenant-specific operations straightforward.
While it makes cross-tenant queries more complex, it's often the cleanest solution for multi-tenant systems where data isolation is important.

<Image
  src="blogs/mnw_125/database_sharding.png"
  width={2945}
  height={797}
  alt="Example of database sharding for scaling a monolith."
/>

## Caching

Caching is one of the most effective ways to improve your system's performance.
A well-implemented caching strategy can dramatically reduce database load and improve response times by storing frequently accessed data in memory.

Modern caching happens at multiple levels.
Browser caching reduces unnecessary network requests.
CDN caching brings your content closer to users.
[**Application-level caching**](caching-in-aspnetcore-improving-application-performance) with tools like Redis stores frequently accessed data in memory.
Database query caching reduces expensive computations.

The key to effective caching is understanding your data access patterns.
Frequently read, rarely changed data benefits most from caching.
Tools like Redis and Memcached excel at storing such data in memory, providing sub-millisecond access times.
Cloud providers offer managed caching services like Azure Cache for Redis,
handling the operational complexity of maintaining a distributed cache.

## Message Queues

Message queues are a powerful tool for scaling your monolith.
They let you defer time-consuming operations and distribute work across multiple processors.
This keeps your API responsive while handling heavy tasks in the background.

Message queues transform your system's behavior under load.
Instead of processing everything synchronously, you can queue work for later.
This pattern works especially well for operations like:

- Processing uploaded files
- Sending emails and notifications
- Generating reports
- Updating search indexes
- Running batch operations

The real power of message queues lies in their ability to handle traffic spikes.
When your system gets hit with a surge of requests, queues act as a buffer.
They let you accept work at peak rates but process it at a sustainable pace.

## Summary

Scaling a monolith isn't about choosing between vertical scaling, horizontal scaling, caching, or any single approach.
It's about using the right tool at the right time.
Start with the simplest solution that solves your immediate problem, then add complexity only when needed.

A practical scaling journey often looks like this:

1. Optimize your code and database queries
2. Add caching where it matters most
3. Scale vertically until it's no longer cost-effective
4. Move to horizontal scaling for better redundancy
5. Implement message queues for background work
6. Consider database sharding when data size demands it

A well-designed monolith can handle significant load with just a subset of these techniques.
The key is to understand your system's actual bottlenecks and address them specifically.

If you're interested in building maintainable, scalable monoliths, my [**Modular Monolith Architecture**](/modular-monolith-architecture) course dives deeper into these concepts.
You'll learn how to structure your code for long-term maintainability and scale.

Remember: don't let perfect be the enemy of good.
Start with the simplest solution that could work, measure everything, and scale what's necessary.
A well-designed monolith can take you further than you might think.

That's all for today. Hope this was helpful.

---



<Sponsorships>

Transform your database performance with [**RavenDB**](https://ravendb.net/ravendb-vs-mongodb?utm_source=influencer_mj&utm_medium=newsletter&utm_campaign=slot1):
Struggling with database bottlenecks and slow queries?
RavenDB is a lightning-fast document database with a distributed architecture that scales effortlessly to meet your needs.
[**Try RavenDB today**](https://ravendb.net/ravendb-vs-mongodb?utm_source=influencer_mj&utm_medium=newsletter&utm_campaign=slot1).

[**AI Agents are the future of AI innovation**](https://fnf.dev/298o5bG).
And APIs will be the key to their success.
Postman CEO Abhinav Asthana predicts that AI agents could increase API utility by 10X-100X and offers some
advice for how to optimize your APIs for agentic integration in [**this insightful article**](https://fnf.dev/298o5bG).

</Sponsorships>

I've been experimenting with different approaches to integrating LLMs into .NET apps,
and I want to share what I've learned about using `Microsoft.Extensions.AI`.

Large Language Models (LLMs) have revolutionized how we approach AI-powered applications.
While many developers are familiar with cloud-based solutions like OpenAI's GPT models,
running LLMs locally has become increasingly accessible thanks to projects like [Ollama](https://ollama.com/).

In this article, we'll explore how to use LLMs in .NET applications using `Microsoft.Extensions.AI`,
a powerful abstraction that extends the [Semantic Kernel](https://github.com/microsoft/semantic-kernel) SDK.

## Understanding the Building Blocks

### Large Language Models (LLMs)

LLMs are deep learning models trained on vast amounts of data, capable of understanding and generating human-like text.
These models can perform various tasks such as text completion, summarization, classification, and engaging in conversation.
While traditionally accessed through cloud APIs, recent advances have made it possible to run them locally on standard hardware.

<figure>
  <Image
    src="blogs/mnw_124/large_language_models.png"
    width={1336}
    height={666}
    alt="Timeline of large language models."
  />
  <figcaption>
    Source: [Weights &
    Biases](https://wandb.ai/vincenttu/blog_posts/reports/A-Survey-of-Large-Language-Models--VmlldzozOTY2MDM1)
  </figcaption>
</figure>

### Ollama

Ollama is an open-source project that simplifies running LLMs locally.
It provides a Docker container that can run various open-source models like Llama,
making it easy to experiment with AI without depending on cloud services.
Ollama handles model management and optimization and provides a simple API for interactions.

### Microsoft.Extensions.AI

[Microsoft.Extensions.AI](https://www.nuget.org/packages/Microsoft.Extensions.AI) is a library that provides a unified interface for working with LLMs in .NET applications.
Built on top of Microsoft's Semantic Kernel, it abstracts away the complexity of different LLM implementations,
allowing developers to switch between providers (like Ollama, Azure, or OpenAI) without changing application code.

## Getting Started

Before diving into the examples, here's what you need to run LLMs locally:

1. Docker running on your machine
2. Ollama container running with the `llama3` model:

```bash
# Pull the Ollama container
docker run --gpus all -d -v ollama_data:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

# Pull the llama3 model
docker exec -it ollama ollama pull llama3
```

3. A few NuGet packages (I built this using a .NET 9 console application):

```powershell
Install-Package Microsoft.Extensions.AI # The base AI library
Install-Package Microsoft.Extensions.AI.Ollama # Ollama provider implementation
Install-Package Microsoft.Extensions.Hosting # For building the DI container
```

## Simple Chat Completion

Let's start with a basic example of chat completion.
Here's the minimal setup:

```csharp {3,7,9}
var builder = Host.CreateApplicationBuilder();

builder.Services.AddChatClient(new OllamaChatClient(new Uri("http://localhost:11434"), "llama3"));

var app = builder.Build();

var chatClient = app.Services.GetRequiredService<IChatClient>();

var chatCompletion = await chatClient.CompleteAsync("What is .NET? Reply in 50 words max.");

Console.WriteLine(chatCompletion.Message.Text);
```

Nothing fancy here - we're just setting up dependency injection and asking a simple question.
If you're used to using raw API calls, you'll notice how clean this feels.

The `AddChatClient` extension method registers the chat client with the DI container.
This allows you to inject `IChatClient` into your services and interact with LLMs using a simple API.
The implementation uses the `OllamaChatClient` to communicate with the Ollama container running locally.

## Implementing Chat with History

Building on the previous example, we can create an interactive chat that maintains conversation history.
This is useful for context-aware interactions and real-time chat applications.
All we need is a `List<ChatMessage` to store the chat history:

```csharp {11}
var chatHistory = new List<ChatMessage>();

while (true)
{
   Console.WriteLine("Enter your prompt:");
   var userPrompt = Console.ReadLine();
   chatHistory.Add(new ChatMessage(ChatRole.User, userPrompt));

   Console.WriteLine("Response from AI:");
   var chatResponse = "";
   await foreach (var item in chatClient.CompleteStreamingAsync(chatHistory))
   {
       // We're streaming the response, so we get each message as it arrives
       Console.Write(item.Text);
       chatResponse += item.Text;
   }
   chatHistory.Add(new ChatMessage(ChatRole.Assistant, chatResponse));
   Console.WriteLine();
}
```

The cool part here is the streaming response - you get that nice, gradual text appearance like in ChatGPT.
We're also maintaining chat history, which lets the model understand context from previous messages, making conversations feel more natural.

## Getting Practical: Article Summarization

Let's try something more useful - automatically summarizing articles.
I've been using this to process blog posts:

```csharp
var posts = Directory.GetFiles("posts").Take(5).ToArray();
foreach (var post in posts)
{
   string prompt = $$"""
         You will receive an input text and the desired output format.
         You need to analyze the text and produce the desired output format.
         You not allow to change code, text, or other references.

         # Desired response

         Only provide a RFC8259 compliant JSON response following this format without deviation.

         {
            "title": "Title pulled from the front matter section",
            "summary": "Summarize the article in no more than 100 words"
         }

         # Article content:

         {{File.ReadAllText(post)}}
         """;

   var chatCompletion = await chatClient.CompleteAsync(prompt);
   Console.WriteLine(chatCompletion.Message.Text);
   Console.WriteLine(Environment.NewLine);
}
```

Pro tip: Being specific about the output format (like requesting [RFC8259](https://datatracker.ietf.org/doc/html/rfc8259) compliant JSON) helps get consistent results.
I learned this the hard way after dealing with occasionally malformed responses!

## Taking It Further: Smart Categorization

Here's where it gets really interesting - we can get strongly typed responses directly from our LLM:

```csharp {29}
class PostCategory
{
    public string Title { get; set; } = string.Empty;
    public string[] Tags { get; set; } = [];
}

var posts = Directory.GetFiles("posts").Take(5).ToArray();
foreach (var post in posts)
{
    string prompt = $$"""
          You will receive an input text and the desired output format.
          You need to analyze the text and produce the desired output format.
          You not allow to change code, text, or other references.

          # Desired response

          Only provide a RFC8259 compliant JSON response following this format without deviation.

          {
             "title": "Title pulled from the front matter section",
             "tags": "Array of tags based on analyzing the article content. Tags should be lowercase."
          }

          # Article content:

          {{File.ReadAllText(post)}}
          """;

    var chatCompletion = await chatClient.CompleteAsync<PostCategory>(prompt);

    Console.WriteLine(
      $"{chatCompletion.Result.Title}. Tags: {string.Join(",",chatCompletion.Result.Tags)}");
}
```

The strongly typed approach provides compile-time safety and better IDE support, making it easier to maintain and refactor code that interacts with LLM responses.

## Flexibility with Different LLM Providers

One of the key advantages of `Microsoft.Extensions.AI` is support for different providers.
While our examples use Ollama, you can easily switch to other providers:

```csharp
// Using Azure OpenAI
builder.Services.AddChatClient(new AzureOpenAIClient(
        new Uri("AZURE_OPENAI_ENDPOINT"),
        new DefaultAzureCredential())
            .AsChatClient());

// Using OpenAI
builder.Services.AddChatClient(new OpenAIClient("OPENAI_API_KEY").AsChatClient());
```

This flexibility allows you to:

- Start development with local models
- Move to production with cloud providers
- Switch between providers without changing application code
- Mix different providers for different use cases (categorization, image recognition, etc.)

## Takeaway

`Microsoft.Extensions.AI` makes it very simple to integrate LLMs into .NET applications.
Whether you're building a chat interface, processing documents, or adding AI-powered features to your application,
the library provides a clean, consistent API that works across different LLM providers.

I've only scratched the surface here.
Since integrating this into my projects, I've found countless uses:

- Automated content moderation for user submissions
- Automated support ticket categorization
- Content summarization for newsletters

I'm also planning a small side project that will use LLMs to process images from a camera feed.
The idea is to detect anything unusual and trigger alerts in real-time.

What are you planning to build with this?
I'd love to hear about your projects and experiences.
The AI space is moving fast, but with tools like `Microsoft.Extensions.AI`, we can focus on building features rather than wrestling with infrastructure.

Good luck out there, and see you next week.

---



<Sponsorships>

Introducing [**Depot Cache**](https://fnf.dev/4gURHAm), the powerful way to make incremental builds
up to 10x faster with tools like Bazel, Turborepo, Gradle, Pants, and sccache.
Run it anywhere—from CI/CD pipelines to local development—and say goodbye to rebuilding from scratch.
To celebrate the launch, Depot is offering your [**first month on any plan completely free—sign up here**](https://fnf.dev/4gURHAm).

Postgres in Azure got easier.
Deploy a [**Neon database**](https://fyi.neon.tech/azureportal) from the Azure portal - it's ready in seconds, it autoscales, has instant dev/test environments, integrates with GitHub.
[**Did we mention there's a generous Free Plan?**](https://fyi.neon.tech/azureportal)

</Sponsorships>

Writing tests is a crucial part of my daily work.
Over the years, I've learned that good tests can make or break a project.

One project I worked on remains the best example of this.
It was a large, complex system with many moving parts.
We had a requirement that code coverage must be above 90%.

Code coverage doesn't directly translate to good tests, but it's a good starting point.
It's up to you to write quality tests that cover the most critical parts of your system.

Today, I want to share my approach to testing Clean Architecture use cases in .NET.

## Why Testing Matters

I've seen many projects fail because of poor testing practices.
The codebase grows, changes become risky, and developers lose confidence in their deployments.
This is especially true for Clean Architecture projects, where we need to ensure our use cases work correctly.

Testing isn't just about catching bugs.
It's about having confidence in your code.
When I make changes, I want to know immediately if I've broken something.
Good tests give me that confidence.

## Understanding Different Testing Approaches

Before diving into the specific examples, let's talk about testing types.
In my experience, there are three main types of tests you'll write:

1. **Unit tests** focus on testing individual components in isolation.
   They're fast, reliable, and help you catch issues early.
   I write these tests first, and they make up the majority of my test suite.

2. [**Integration tests**](testcontainers-integration-testing-using-docker-in-dotnet) verify that different components work together correctly.
   They're slower but essential for testing database operations or external services.

3. [**End-to-end tests**](testing-modular-monoliths-system-integration-testing) check the entire system flow.
   They're the slowest but provide confidence that everything works together.

For this article, we'll focus on unit tests. They're the foundation of a solid test suite and the most common type you'll write.

## Breaking Down Our Use Case

Looking at our `ReserveBookingCommandHandler` class, we have a typical Clean Architecture use case.
It handles apartment booking reservations with several business rules:

1. The apartment must exist
2. The booking dates must not overlap with existing bookings
3. A new booking should be created if all checks pass

This is a perfect example for unit testing because it has clear inputs, outputs, and dependencies we can mock.

```csharp
internal sealed class ReserveBookingCommandHandler(
    IApartmentRepository apartmentRepository,
    IBookingRepository bookingRepository,
    IDateTimeProvider dateTimeProvider) : ICommandHandler<ReserveBookingCommand, Guid>
{
    public async Task<Result<Guid>> Handle(
        ReserveBookingCommand request,
        CancellationToken cancellationToken)
    {
        var apartment = await apartmentRepository.GetByIdAsync(request.ApartmentId, cancellationToken);

        if (apartment is null)
        {
            return Result.Failure<Guid>(ApartmentErrors.NotFound);
        }

        var duration = DateRange.Create(request.StartDate, request.EndDate);

        if (await bookingRepository.IsOverlappingAsync(apartment, duration, cancellationToken))
        {
            return Result.Failure<Guid>(BookingErrors.Overlap);
        }

        var booking = Booking.Create(
            apartment,
            duration,
            dateTimeProvider.UtcNow);

        bookingRepository.Add(booking);

        return booking.Id;
    }
}
```

## Setting Up Our Test Environment

The test class setup shows the standard approach I use for all my handler tests. Let's break it down:

```csharp {16}
public class ReserveBookingCommandHandlerTests
{
    private readonly ReserveBookingCommandHandler _handler;
    private readonly IApartmentRepository _apartmentRepository;
    private readonly IBookingRepository _bookingRepository;
    private readonly IDateTimeProvider _dateTimeProvider;

    private static readonly Guid ApartmentId = Guid.NewGuid();
    private static readonly DateTime UtcNow = DateTime.UtcNow;

    public ReserveBookingCommandHandlerTests()
    {
        _apartmentRepository = Substitute.For<IApartmentRepository>();
        _bookingRepository = Substitute.For<IBookingRepository>();
        _dateTimeProvider = Substitute.For<IDateTimeProvider>();
        _dateTimeProvider.UtcNow.Returns(UtcNow);

        _handler = new ReserveBookingCommandHandler(
            _apartmentRepository,
            _bookingRepository,
            _dateTimeProvider);
    }
}
```

I'm using [NSubstitute](https://nsubstitute.github.io/) to create mocks of our dependencies.
Each test starts with fresh mocks, preventing test interference.
The static fields provide consistent values across all tests.

Notice how I mock `IDateTimeProvider`.
This is crucial for testing time-dependent code.
Never use `DateTime.UtcNow` directly in your production code - it makes testing much harder.

## Testing the Not Found Scenario

Our first test verifies the behavior when an apartment doesn't exist:

```csharp {10,11}
[Fact]
public async Task Handle_WhenApartmentDoesNotExist_ShouldReturnNotFoundError()
{
    // Arrange
    var command = new ReserveBookingCommand(
        ApartmentId,
        new DateOnly(2024, 1, 1),
        new DateOnly(2024, 1, 5));

    _apartmentRepository.GetByIdAsync(ApartmentId, Arg.Any<CancellationToken>())
        .Returns((Apartment?)null);

    // Act
    var result = await _handler.Handle(command, default);

    // Assert
    result.IsFailure.Should().BeTrue();
    result.Error.Should().Be(ApartmentErrors.NotFound);
}
```

This test follows the **Arrange-Act-Assert** pattern:

1. Arrange: Set up the command and mock the repository to return `null`
2. Act: Call the handler
3. Assert: Verify we get the correct error

I use [FluentAssertions](https://fluentassertions.com/) because it provides clear, readable assertions and better error messages than the standard Assert class.

## Handling Booking Conflicts

The overlap test ensures we can't double-book apartments:

```csharp {10-14}
[Fact]
public async Task Handle_WhenBookingOverlaps_ShouldReturnOverlapError()
{
    // Arrange
    var command = new ReserveBookingCommand(
        ApartmentId,
        new DateOnly(2024, 1, 1),
        new DateOnly(2024, 1, 5));

    var apartment = new Apartment { Id = ApartmentId };
    _apartmentRepository.GetByIdAsync(ApartmentId, Arg.Any<CancellationToken>())
        .Returns(apartment);
    _bookingRepository.IsOverlappingAsync(apartment, Arg.Any<DateRange>(), Arg.Any<CancellationToken>())
        .Returns(true);

    // Act
    var result = await _handler.Handle(command, default);

    // Assert
    result.IsFailure.Should().BeTrue();
    result.Error.Should().Be(BookingErrors.Overlap);
}
```

Here, we verify the overlap check works correctly.
Notice how we:

1. Mock the apartment repository to return a valid apartment
2. Mock the booking repository to indicate an overlap
3. Verify we get the overlap error

## Testing Successful Bookings

The happy path test ensures everything works when all conditions are met:

```csharp {21-24}
[Fact]
public async Task Handle_WhenValidRequest_ShouldCreateBooking()
{
    // Arrange
    var command = new ReserveBookingCommand(
        ApartmentId,
        new DateOnly(2024, 1, 1),
        new DateOnly(2024, 1, 5));

    var apartment = new Apartment { Id = ApartmentId };
    _apartmentRepository.GetByIdAsync(ApartmentId, Arg.Any<CancellationToken>())
        .Returns(apartment);
    _bookingRepository.IsOverlappingAsync(apartment, Arg.Any<DateRange>(), Arg.Any<CancellationToken>())
        .Returns(false);

    // Act
    var result = await _handler.Handle(command, default);

    // Assert
    result.IsSuccess.Should().BeTrue();
    await _bookingRepository.Received(1)
        .Add(Arg.Is<Booking>(b =>
            b.Id == result.Value &&
            b.ApartmentId == ApartmentId));
}
```

This test is more complex because we need to:

1. Set up multiple mocks
2. Verify the success result
3. Check that the booking was added with correct properties

NSubstitute's `Received()` method lets us verify the `Add` method was called **exactly once** with the right booking.

## Verifying Exception Handling

Testing exception scenarios is crucial for robust code:

```csharp {14,20}
[Fact]
public async Task Handle_WhenRepositoryThrowsOverlapException_ShouldPropagateException()
{
    // Arrange
    var command = new ReserveBookingCommand(
        ApartmentId,
        new DateOnly(2024, 1, 1),
        new DateOnly(2024, 1, 5));

    var apartment = new Apartment { Id = ApartmentId };
    _apartmentRepository.GetByIdAsync(ApartmentId, Arg.Any<CancellationToken>())
        .Returns(apartment);
    _bookingRepository.IsOverlappingAsync(apartment, Arg.Any<DateRange>(), Arg.Any<CancellationToken>())
        .Throws<BookingOverlapException>();

    // Act
    var act = () => _handler.Handle(command, default);

    // Assert
    await act.Should().ThrowAsync<BookingOverlapException>();
}
```

This test ensures exceptions propagate correctly. We:

1. Set up the scenario
2. Make the repository throw an exception
3. Verify the exception bubbles up

FluentAssertions makes testing async exceptions clean and readable.

## Understanding Test Coverage Limitations

When we look back at our booking overlap test, there's an important distinction to make.
Our unit test verifies that our command handler behaves correctly when the booking repository reports an overlap.
However, it doesn't verify that the overlap detection logic itself works correctly.

Consider what we're actually testing:

```csharp
_bookingRepository.IsOverlappingAsync(apartment, Arg.Any<DateRange>(), Arg.Any<CancellationToken>())
    .Returns(true);
```

We're simply telling our mock repository to return `true`.
This gives us confidence that our command handler correctly handles the overlap scenario,
but it does not tell us whether our actual overlap detection logic works correctly.

This is where integration tests become essential.
An integration test for this scenario would:

- Insert real bookings into a test database
- Attempt to create overlapping bookings
- Verify that the overlap detection works with real data

The combination of unit and integration tests provides complete coverage:

- Unit tests verify the business logic flow
- Integration tests verify the actual overlap detection logic

This example highlights why we need different types of tests.
Unit tests are excellent for verifying behavior and logic flows,
but they can't verify the correctness of complex business rules that depend on real data interactions.

## Summary

Unit testing Clean Architecture use cases requires careful thought about dependencies and behavior.
Here are the key points:

- Mock all external dependencies
- Test both success and failure scenarios
- Verify exception handling logic
- Use descriptive test names
- Follow the Arrange-Act-Assert pattern

Good tests act as documentation.
They show how the code should behave and catch issues before they reach production.
Invest time in writing good tests - your future self will thank you.

Want to dive deeper into testing Clean Architecture applications?
I cover this and much more in my [**Pragmatic Clean Architecture**](/pragmatic-clean-architecture) course.
You'll learn how to write effective unit tests, integration tests, and end-to-end tests that give you real confidence in your system.

Writing quality tests is a skill that improves with practice and understanding.
Take the time to write them well.

That's all for today.

See you next week.

---



<Sponsorships>

As the year wraps up, it's clear [**API collaboration**](https://fnf.dev/4gjsqA6) took a big step forward.
The Postman VS Code extension, handy templates, and the Vault for secure data have all played their part.
[**Catch it all in Postman's December drop**](https://fnf.dev/4gjsqA6).
Here's to progress — and to working together again in the new year!

Stefan Djokic is launching his YouTube channel and building [**The CodeMan Community**](https://www.skool.com/thecodeman-community-2911/about) - your hub for .NET content, mini-courses, and expert advice!
The first 100 members get in for just $4/month!
[**Join now and get a free ebook**](https://www.skool.com/thecodeman-community-2911/about)!

</Sponsorships>

"Your task is to rewrite this system. It powers our entire operation. Oh, and it's written in APL."

That's how my journey with this legacy rewrite began.
For those unfamiliar with [APL](<https://en.wikipedia.org/wiki/APL_(programming_language)>),
it's a programming language from the 1960s known for its unique mathematical notation and array manipulation capabilities.
Finding developers who know APL today is about as easy as finding a floppy disk drive in a modern computer.

The system has grown over four decades.
It started as a simple inventory management tool and evolved into a comprehensive [ERP](https://en.wikipedia.org/wiki/Enterprise_resource_planning) system.
More than 460+ database tables.
Countless business rules embedded in the code.
Complex integrations with every part of the business process.
The system is the backbone of a manufacturing operation, that generates over $10 million in annual revenue.

Our mission was clear but daunting: modernize this system using .NET, PostgreSQL, and React.

The catch?
The business needed to keep running during the transition.
No downtime.
No data loss.
No disruption to daily operations.

This wasn't just a technical challenge.
It was a lesson in managing complexity, understanding legacy business processes, and navigating organizational dynamics.

So here's that story and the lessons learned.

## Initial State: Understanding the Legacy

The first challenge was understanding how this massive system actually worked.
The codebase had grown organically over four decades, maintained by a single development team.
They were now in their 60s and looking to retire.

Walking into the first codebase review was like opening a time capsule.
APL's concise syntax meant that complex business logic could be written in just a few lines.
Beautiful, if you could read it.
Terrifying, if you couldn't.
And most of us couldn't.

The original team was invaluable during the knowledge transfer.
They knew every quirk, every special case, every business rule that had been added over the decades.
But there's only so much you can learn from conversations.
Documentation was sparse.
What existed was outdated.
The real documentation was in the heads of the original developers.

We spent weeks mapping the system's functionality:

- The core manufacturing process was spread across 50+ tables with complex interdependencies
- Inventory management touched nearly every part of the system
- Custom reporting tools have been built over decades to meet specific business needs
- Integration points with external components were handled through a maze of stored procedures

Tables that started with basic schemas had grown to include hundreds of columns.
Some columns were no longer used but couldn't be removed because no one was sure if some obscure report still needed them.

What made this particularly challenging was the disconnect between the business processes and their technical implementation.
The business would describe a simple workflow, but the technical implementation would reveal layers of complexity added over years of edge cases and special requirements.

We needed a systematic approach to understanding this beast.
We started by mapping business processes and their corresponding technical implementations.
This helped us identify the core domains that would later influence our modular architecture.
More importantly, it helped us understand the true scope of what we were dealing with.

## The Product vs. Engineering Conflict

Management wanted quick wins.
They pushed us to start with the simplest components.
This created tension between product management and the development team.

Product management's perspective was straightforward: show progress to the business.
They needed visible results to justify the investment in the rewrite.
The business was spending significant money, and they wanted to see returns quickly.

The development team saw a different reality.
We knew that starting with peripheral features meant building on shaky ground.
The core business logic would remain in the legacy system, making every integration point more complex.
This technical debt would compound over time.

As a technical lead, I strongly opposed this approach.
My argument was simple: the core manufacturing process was the heart of the system.
Every peripheral feature depended on it.
By postponing its migration, we created a tangled web of dependencies between old and new systems.
Each new feature we migrated would need complex synchronization with the legacy core.
We were building on quicksand.

I advocated for focusing on the core domain first.
Yes, it would take longer to show the first results.
But it would create a solid foundation for everything that followed.
The business would have to wait longer for visible progress, but the overall migration would be faster and more reliable.

Neither side was wrong in their objectives.
Product management had valid concerns about showing progress.
The development team had valid concerns about technical sustainability.
But this misalignment led to compromises that impacted the project timeline.
To this day, I believe we would have finished the migration sooner if we had started with the core business logic.

## Software Architecture: Building for the Future

During the discovery phase, we identified distinct business domains within the system.
This led us to implement a [**modular monolith architecture**](what-is-a-modular-monolith).
Each module would be self-contained but able to communicate with others through a shared event bus:

<Image
  src="blogs/mnw_122/legacy_system_with_modular_monolith.png"
  width={4453}
  height={2381}
  alt="Modular monolith architecture with legacy system and a message broker for communication."
/>

Key architectural decisions:

1. [**Modular monolith**](monolith-to-microservices-how-a-modular-monolith-helps):
   Each module represented a distinct business domain.
   This provided a clear path to potential future microservices if needed.

2. [**Asynchronous communication**](modular-monolith-communication-patterns):
   Modules communicated through events using RabbitMQ.
   This reduced coupling and improved system resilience.

3. [**Shared database with boundaries**](modular-monolith-data-isolation):
   While all modules used the same PostgreSQL database, each had its own set of tables and schemas.
   This helped us maintain logical separation.

4. [**Cloud-ready design**](dotnet-aspire-a-game-changer-for-cloud-native-development):
   The system was deployed to AWS using containerization.
   A Jenkins pipeline enabled deployments to multiple environments in minutes.

## The Data Sync Challenge

The two-way data synchronization was more complex than initially anticipated.
Here's why we couldn't use existing [change data capture](https://en.wikipedia.org/wiki/Change_data_capture) (CDC) solutions like [Debezium](https://debezium.io/):

1. **Complex transformations**:
   Many legacy tables required data from multiple new tables.
   This wasn't a simple one-to-one mapping that CDC tools excel at.

2. **Business logic in sync**:
   The sync process needed to apply business rules during transformation.
   This went beyond what most replication tools provide.

3. **Bidirectional requirements**:
   We needed to sync both ways while preventing infinite loops.
   The legacy system remained the source of truth for non-migrated components.

<Image
  src="blogs/mnw_122/data_sync.png"
  width={3380}
  height={1732}
  alt="Data sync flow between modern and legacy system."
/>

We built a custom solution using RabbitMQ for message transport.
While this worked for us, the lesson remains: evaluate existing tools thoroughly before building custom solutions.
Even if you can't use them entirely, you might learn valuable patterns from their approaches.

## Key Technical Lessons

1. **Modular architecture pays off**:
   The modular monolith approach made the system easier to understand and maintain.
   Each module had clear boundaries and responsibilities.

2. **Invest in deployment automation**:
   The CI/CD pipeline was crucial.
   It allowed us to deploy confidently and frequently, reducing the risk of each change.

3. **Message-based integration**:
   Async communication between modules provided the flexibility needed for the gradual migration.

4. **Data sync complexity**:
   Don't underestimate the complexity of data synchronization in legacy migrations.
   Whether using existing tools or building custom solutions, this will be a major challenge.

## The Human Factor

Technical challenges are only part of the story.
The success of legacy rewrites depends heavily on managing different stakeholders:

1. Product Management needs to see progress
2. Development teams need time to do things right
3. The business needs to keep running
4. The legacy team needs to transfer knowledge

Finding the right balance between these competing needs can be tricky.

We found several approaches that helped:

- Regular stakeholder meetings where each group could voice concerns
- Transparent project tracking visible to all parties
- Clear communication about technical decisions and their business impact
- Celebration of both technical and business milestones
- Documentation of both technical and institutional knowledge

I can't stress enough how important it was to document the knowledge acquired over four decades of operating the legacy system.
When the original team retired, we had a comprehensive set of documents that explained every business rule and every edge case.

## Results That Matter

Four years later, the system is thriving.
The cloud infrastructure provides reliability and scalability.
The [**modular monolith architecture**](/modular-monolith-architecture) makes it maintainable.
The automated deployment pipeline enables rapid updates.

But the journey taught us valuable lessons about balancing technical needs with business pressures.
Success in legacy rewrites requires more than just technical excellence.
It requires understanding the business domain, managing stakeholder expectations, and making pragmatic architectural decisions.

Software architecture matters, but so does the human factor. Plan for both.

Thanks for reading.

And stay awesome!

---



<Sponsorships>

It's been a big year for API collaborations!
The [**Postman VS Code extension**](https://fnf.dev/4gjsqA6) helped us integrate our workflows, new templates got us started when we were stuck,
and the Postman Vault kept our data secure.
[**Postman's December drop**](https://fnf.dev/4gjsqA6) has all the details on these and more.

[**Get 1 Dometrain Course for free**](https://dometrain.com/courses?ref=milan-jovanovic&promo=newsletter-20241221&coupon_code=MILAN01)!
Dometrain is a learning platform by Nick Chapsas.
With more than 265.000 enrollments, there is no better place to invest in your learning as a .NET engineer.
Use code MILAN01 to get 20% off any course, and add more than one item to your basket to get one for free!
[**Get your free course**](https://dometrain.com/courses?ref=milan-jovanovic&promo=newsletter-20241221&coupon_code=MILAN01)!

</Sponsorships>

Most ASP.NET Core applications need to handle [**background processing**](running-background-tasks-in-asp-net-core) -
from sending reminder emails to running cleanup tasks.
While there are many ways to implement background jobs, [Quartz.NET](https://www.quartz-scheduler.net/)
stands out with its robust scheduling capabilities, persistence options, and production-ready features.

In this article, we'll look at:

- Setting up Quartz.NET with ASP.NET Core and proper observability
- Implementing both on-demand and recurring jobs
- Configuring persistent storage with PostgreSQL
- Handling job data and monitoring execution

Let's start with the basic setup and build our way up to a production-ready configuration.

## Setting Up Quartz With ASP.NET Core

First, let's set up Quartz with proper instrumentation.

We'll need to install some NuGet packages:

```powershell
Install-Package Quartz.Extensions.Hosting
Install-Package Quartz.Serialization.Json

# This might be in prerelease
Install-Package OpenTelemetry.Instrumentation.Quartz
```

Next, we'll configure the Quartz services and OpenTelemetry instrumentation and start the scheduler:

```csharp {15}
builder.Services.AddQuartz();

// Add Quartz.NET as a hosted service
builder.Services.AddQuartzHostedService(options =>
{
    options.WaitForJobsToComplete = true;
});

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        tracing
            .AddHttpClientInstrumentation()
            .AddAspNetCoreInstrumentation()
            .AddQuartzInstrumentation();
    })
    .UseOtlpExporter();
```

This is all we need at the start.

## Defining and Scheduling Jobs

To define a [**background job**](scheduling-background-jobs-with-quartz-net), you have to implement the `IJob` interface.
All job implementations run as scoped services, so you can inject dependencies as needed.
Quartz allows you to pass data to a job using the `JobDataMap` dictionary.
It's recommended to only use primitive types for job data to avoid serialization issues.

When executing the job, there are a few ways to fetch job data:

- `JobDataMap` - a dictionary of key-value pairs
  - `JobExecutionContext.JobDetail.JobDataMap` - job-specific data
  - `JobExecutionContext.Trigger.TriggerDataMap` - trigger-specific data
- `MergedJobDataMap` - combines job data with trigger data

It's a best practice to use `MergedJobDataMap` to retrieve job data.

```csharp
public class EmailReminderJob(ILogger<EmailReminderJob> logger, IEmailService emailService) : IJob
{
    public const string Name = nameof(EmailReminderJob);

    public async Task Execute(IJobExecutionContext context)
    {
        // Best practice: Prefer using MergedJobDataMap
        var data = context.MergedJobDataMap;

        // Get job data - note that this isn't strongly typed
        string? userId = data.GetString("userId");
        string? message = data.GetString("message");

        try
        {
            await emailService.SendReminderAsync(userId, message);

            logger.LogInformation("Sent reminder to user {UserId}: {Message}", userId, message);
        }
        catch (Exception ex)
        {
            logger.LogError(ex, "Failed to send reminder to user {UserId}", userId);

            // Rethrow to let Quartz handle retry logic
            throw;
        }
    }
}
```

One thing to note: `JobDataMap` isn't strongly typed. This is a limitation we have to live with, but we can mitigate it by:

1. Using constants for key names
2. Validating data early in the `Execute` method
3. Creating wrapper services for job scheduling

Now, let's discuss scheduling jobs.

Here's how to schedule one-time reminders:

```csharp {9}
public record ScheduleReminderRequest(
    string UserId,
    string Message,
    DateTime ScheduleTime
);

// Schedule a one-time reminder
app.MapPost("/api/reminders/schedule", async (
    ISchedulerFactory schedulerFactory,
    ScheduleReminderRequest request) =>
{
    var scheduler = await schedulerFactory.GetScheduler();

    var jobData = new JobDataMap
    {
        { "userId", request.UserId },
        { "message", request.Message }
    };

    var job = JobBuilder.Create<EmailReminderJob>()
        .WithIdentity($"reminder-{Guid.NewGuid()}", "email-reminders")
        .SetJobData(jobData)
        .Build();

    var trigger = TriggerBuilder.Create()
        .WithIdentity($"trigger-{Guid.NewGuid()}", "email-reminders")
        .StartAt(request.ScheduleTime)
        .Build();

    await scheduler.ScheduleJob(job, trigger);

    return Results.Ok(new { scheduled = true, scheduledTime = request.ScheduleTime });
})
.WithName("ScheduleReminder")
.WithOpenApi();

```

The endpoint schedules one-time email reminders using Quartz.
It creates a job with user data, sets up a trigger for the specified time, and schedules them together.
The `EmailReminderJob` receives a unique identity in the `email-reminders` group.

Here's a sample request you can use to test this out:

```
POST /api/reminders/schedule
{
    "userId": "user123",
    "message": "Important meeting!",
    "scheduleTime": "2024-12-17T15:00:00"
}
```

## Scheduling Recurring Jobs

For recurring background jobs, you can use [cron schedules](https://www.quartz-scheduler.net/documentation/quartz-3.x/tutorial/crontriggers.html):

```csharp {4,9,27}
public record RecurringReminderRequest(
    string UserId,
    string Message,
    string CronExpression
);

// Schedule a recurring reminder
app.MapPost("/api/reminders/schedule/recurring", async (
    ISchedulerFactory schedulerFactory,
    RecurringReminderRequest request) =>
{
    var scheduler = await schedulerFactory.GetScheduler();

    var jobData = new JobDataMap
    {
        { "userId", request.UserId },
        { "message", request.Message }
    };

    var job = JobBuilder.Create<EmailReminderJob>()
        .WithIdentity($"recurring-{Guid.NewGuid()}", "recurring-reminders")
        .SetJobData(jobData)
        .Build();

    var trigger = TriggerBuilder.Create()
        .WithIdentity($"recurring-trigger-{Guid.NewGuid()}", "recurring-reminders")
        .WithCronSchedule(request.CronExpression)
        .Build();

    await scheduler.ScheduleJob(job, trigger);

    return Results.Ok(new { scheduled = true, cronExpression = request.CronExpression });
})
.WithName("ScheduleRecurringReminder")
.WithOpenApi();
```

Cron triggers are more powerful than simple triggers.
They allow you to define complex schedules like "every weekday at 10 AM" or "every 15 minutes".
Quartz supports cron expressions with seconds, minutes, hours, days, months, and years.

Here's a sample request if you want to test this:

```
POST /api/reminders/schedule/recurring
{
    "userId": "user123",
    "message": "Daily standup",
    "cronExpression": "0 0 10 ? * MON-FRI"
}
```

## Job Persistence Setup

By default, Quartz uses in-memory storage, which means your jobs are lost when the application restarts.
For production environments, you'll want to use a persistent store.
Quartz supports several [database providers](https://www.quartz-scheduler.net/documentation/quartz-3.x/tutorial/job-stores.html),
including SQL Server, PostgreSQL, MySQL, and Oracle.

Let's look at how to set up persistent storage with proper schema isolation:

```csharp {11}
builder.Services.AddQuartz(options =>
{
    options.AddJob<EmailReminderJob>(c => c
        .StoreDurably()
        .WithIdentity(EmailReminderJob.Name));

    options.UsePersistentStore(persistenceOptions =>
    {
        persistenceOptions.UsePostgres(cfg =>
        {
            cfg.ConnectionString = connectionString;
            cfg.TablePrefix = "scheduler.qrtz_";
        },
        dataSourceName: "reminders"); // Database name

        persistenceOptions.UseNewtonsoftJsonSerializer();
        persistenceOptions.UseProperties = true;
    });
});
```

A few important things to note here:

- The `TablePrefix` setting helps organize Quartz tables in your database - in this case, placing them in a dedicated `scheduler` schema
- You'll need to run the appropriate database scripts to create these tables
- Each database provider has its own [setup scripts](https://github.com/quartznet/quartznet/tree/main/database/tables) -
  check the Quartz documentation for your chosen provider

### Durable Jobs

Notice how we're configuring the `EmailReminderJob` with `StoreDurably`?
This is a powerful pattern that lets you define your jobs once and reuse them with different triggers.
Here's how to schedule a stored job:

```csharp
public async Task ScheduleReminder(string userId, string message, DateTime scheduledTime)
{
    var scheduler = await _schedulerFactory.GetScheduler();

    // Reference the stored job by its identity
    var jobKey = new JobKey(EmailReminderJob.Name);

    var trigger = TriggerBuilder.Create()
        .ForJob(jobKey)  // Reference the durable job
        .WithIdentity($"trigger-{Guid.NewGuid()}")
        .UsingJobData("userId", userId)
        .UsingJobData("message", message)
        .StartAt(scheduledTime)
        .Build();

    await scheduler.ScheduleJob(trigger);  // Note: just passing the trigger
}
```

This approach has several benefits:

- Job definitions are centralized in your startup configuration
- You can't accidentally schedule a job that hasn't been properly configured
- Job configurations are consistent across all schedules

## Summary

Getting **Quartz** set up properly in .NET involves more than just adding the NuGet package.

Pay attention to:

1. Proper job definition and data handling with `JobDataMap`
2. Setting up both one-time and recurring job schedules
3. Configuring persistent storage with proper schema isolation
4. Using durable jobs to maintain consistent job definitions

Each of these elements contributes to a reliable background processing system that can grow with your application's needs.
A good example of using background jobs is when you want to [**build asynchronous APIs**](building-async-apis-in-aspnetcore-the-right-way).

Good luck out there, and I'll see you next week.

---



<Sponsorships>

[**Get 1 Dometrain Course for free**](https://dometrain.com/courses?ref=milan-jovanovic&promo=newsletter-20241214&coupon_code=MILAN01)!
Dometrain is a learning platform by Nick Chapsas.
With more than 265.000 enrollments, there is no better place to invest in your learning as a .NET engineer.
Use code MILAN01 to get 20% off any course, and add more than one item to your basket to get one for free!
[**Get your free course**](https://dometrain.com/courses?ref=milan-jovanovic&promo=newsletter-20241214&coupon_code=MILAN01)!

Three things to know about [**Depot**](https://fnf.dev/4gtFbHV):\
• Up to 3x faster: Depot's Ultra Runners make GitHub Actions jobs 3x faster than GitHub's default runners.\
• Ideal for AI workloads, npm install, git clone, and other heavy file operations.\
• **Get one month free on any plan** — [**Sign up**](https://fnf.dev/4gtFbHV) and use code **DEPOTFRIENDS24** at checkout — valid through December 20.

</Sponsorships>

Every article about modular monoliths tells you to use public APIs between modules.
But they rarely tell you why these APIs exist or how to design them properly.

A modular monolith organizes an application into independent modules that have clear boundaries.
The module boundaries are logical and group related business capabilities together.

After building several large-scale modular monoliths, I've learned that public APIs are not just about clean code - they're about controlling chaos.
Let me show you what I mean.

## The Reality of Module Communication

Here's what nobody tells you about public APIs in modular monoliths: they represent intentional coupling points.
Yes, you read that right.
Public APIs don't eliminate coupling - they make it explicit and controllable.

When Module A needs something from Module B, you have three options:

1. Let Module A read directly from Module B's database
2. Let Module A access Module B's internal services
3. Create a public API that explicitly defines what Module A can do

<Image
  src="blogs/mnw_120/module_communication_options.png"
  width={2189}
  height={1462}
  alt="Module communication options: direct database access, calling internal services, calling public API."
/>

The first two options lead to chaos.
I've seen entire systems become unmaintainable because every module was freely accessing the data and services of other modules.

The previous options are examples of synchronous communication between modules.
But you can also implement [**asynchonrous module communication**](modular-monolith-communication-patterns) using messaging.
We have to adjust the technical implementation.
However, modules still have a public API in message contracts.

## Why We Need Public APIs

Public APIs serve three critical purposes:

1. **Contract Definition**: They explicitly state what other modules can and cannot do
2. **Dependency Control**: They force you to think about module dependencies
3. **Change Management**: They provide a stable interface while allowing internal changes

Here's a practical example. Imagine you have an Orders module and a Shipping module.

This is what you want to avoid:

```csharp
public class ShippingService
{
    private readonly OrdersDbContext _ordersDb; // Direct database access

    public async Task ShipOrder(string orderId)
    {
        // Directly reading from another module's database
        var order = await _ordersDb.Orders
            .Include(o => o.Lines)
            .FirstOrDefaultAsync(o => o.Id == orderId);

        // What happens if the Orders module changes its schema?
        // What if it moves to a different database?
    }
}
```

This is what you want to achieve instead:

```csharp
public class ShippingService
{
    private readonly IOrdersModule _orders; // Public API access

    public async Task ShipOrder(string orderId)
    {
        // Using the public API
        var order = await _orders.GetOrderForShippingAsync(orderId);

        // The Orders module can change its internals
        // as long as it maintains this contract
    }
}
```

## Controlling What Gets Exposed

The hardest part of designing public APIs is deciding what to expose.
Here's my rule of thumb:

1. Start with nothing public
2. Expose only what other modules actually need
3. Design the API around use cases, not data

Here's how this looks in practice:

```csharp
public interface IOrdersModule
{
    // Don't expose generic CRUD operations
    // Task<Order> GetOrderAsync(string orderId); // Bad

    // Instead, expose specific use cases
    Task<OrderShippingInfo> GetOrderForShippingAsync(string orderId);
    Task<OrderPaymentInfo> GetOrderForPaymentAsync(string orderId);
    Task<OrderSummary> GetOrderForCustomerAsync(string orderId);
}
```

## Protecting Your Module's Data

Public APIs aren't enough.
You also need to [**protect your module's data**](modular-monolith-data-isolation).
Here's what I've found works:

1. **Separate Schemas**: Each module gets its own database schema.

```sql
CREATE SCHEMA Orders;
CREATE SCHEMA Shipping;

-- Orders module can only access its schema
CREATE USER OrdersUser WITH DEFAULT_SCHEMA = Orders;
GRANT SELECT, INSERT, UPDATE, DELETE ON SCHEMA::Orders TO OrdersUser;

-- Shipping module can only access its schema
CREATE USER ShippingUser WITH DEFAULT_SCHEMA = Shipping;
GRANT SELECT, INSERT, UPDATE, DELETE ON SCHEMA::Shipping TO ShippingUser;
```

We can also lock down the user's access to a given schema to only allow reading and writing data.

2. **Different Connection Strings**: Each module gets its own database user with a respective connection string.

```csharp
builder.Services.AddDbContext<OrdersDbContext>(options =>
    options.UseSqlServer(builder.Configuration.GetConnectionString("OrdersConnection")));

builder.Services.AddDbContext<ShippingDbContext>(options =>
    options.UseSqlServer(builder.Configuration.GetConnectionString("ShippingConnection")));
```

If you want to learn more about this, check out this article about [**using multiple EF Core DbContexts**](using-multiple-ef-core-dbcontext-in-single-application).

3. **Read Models**: Create specific read models for other modules.

```csharp
internal class Order
{
    // Internal domain model with full complexity
}

public class OrderShippingInfo
{
    // Public DTO with only what shipping needs
    public string OrderId { get; init; }
    public Address ShippingAddress { get; init; }
    public List<ShippingItem> Items { get; init; }
}
```

## Dealing with Cross-Cutting Concerns

Some features naturally span multiple modules.
For example, when a customer views their order history, you might need data from the Orders, Shipping, and Payments modules.

Don't try to force this through module APIs.
Instead:

1. Create a separate query model
2. Use event-driven patterns to keep it updated
3. Own it in a dedicated module or one of the existing modules

```csharp
public class OrderHistoryModule
{
    public async Task<CustomerOrderHistory> GetOrderHistoryAsync(string customerId)
    {
        // Read from a dedicated read model that's kept
        // updated through events from other modules
        return await _orderHistoryRepository.GetCustomerHistoryAsync(customerId);
    }
}
```

## Summary

Public APIs in modular monoliths are not about preventing coupling - they're about controlling it.
Every public API is a contract that says: "Yes, these modules are coupled, and this is exactly how they depend on each other."

The goal isn't to eliminate dependencies between modules.
The goal is to make them explicit, controlled, and maintainable.

Get this right, and your modular monolith will be easier to maintain, test, and evolve.
Get it wrong, and you'll end up with a distributed big ball of mud.

Want to master building modular monoliths with clean APIs and event-driven patterns?
Check out my [**Modular Monolith Architecture**](/modular-monolith-architecture) course, where I'll show you how to build maintainable systems
using practical examples from real projects.

That's all for today. Stay awesome, and I'll see you next week.

---



<Sponsorships>

APIs can be frustrating — think inconsistent standards, incomplete docs, and unpredictable errors in production.
[**Postman's latest article**](https://shortclick.link/i6muq1) breaks down common struggles with APIs and how better tools and workflows can help.
[**Read the article**](https://shortclick.link/i6muq1) to learn how to simplify your workflow and tackle common API challenges.

[**Share your development expertise, get rewarded!**](https://ironsoftware.com/partners/affiliate/?utm_source=milan&utm_medium=email&utm_campaign=milan)
Empower developers by sharing your experiences with Iron Software libraries.
Your tutorials can help others find powerful tools and you'll earn exclusive rewards for sharing them!
[**Join the Developer Knowledge Network**](https://ironsoftware.com/partners/affiliate/?utm_source=milan&utm_medium=email&utm_campaign=milan).

</Sponsorships>

I remember the days when managing NuGet packages across multiple projects was a real pain.
You know what I mean - you open a large solution and find out every project uses a different version of the same package.
Not fun!

Let me show you how **Central Package Management** (CPM) in .NET can fix this problem once and for all.

## The Problem We Need to Solve

I often work with solutions that have lots of projects.
It's not uncommon to have solutions with 30 or more projects.
Each one needs similar packages like Serilog or Polly.
Most test projects I create depend on xUnit.
Before CPM, keeping track of package versions was a mess:

- One project uses Serilog `4.1.0`
- Another uses Serilog `4.0.2`
- And somehow, a third one uses Serilog `3.1.1`

This causes real problems.
Different versions can behave differently, leading to weird bugs that are hard to track down.
I've wasted many hours fixing issues caused by version mismatches.

## How Central Package Management Helps

Think of CPM as a control center for all your package versions.
Instead of setting versions in each project, you set them once in one place.
Then, you just reference a package you want to use without specifying the version.
It's that simple.

Here's what you need to use [Central Package Management](https://learn.microsoft.com/en-us/nuget/consume-packages/central-package-management):

- NuGet 6.2 or newer
- .NET SDK 6.0.300 or newer
- If you use Visual Studio, you need version 2022 17.2 or newer

## Setting It Up

Let me show you how to set up CPM.
It's easier than you might think.

1. First, create a file called `Directory.Packages.props` in your solution's main folder:

```xml
<Project>
  <PropertyGroup>
    <ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
  </PropertyGroup>
  <ItemGroup>
    <PackageVersion Include="Newtonsoft.Json" Version="13.0.3" />
    <PackageVersion Include="Serilog" Version="4.1.0" />
    <PackageVersion Include="Polly" Version="8.5.0" />
  </ItemGroup>
</Project>
```

Note the use of `PackageVersion` to define NuGet dependencies.

2. In your project files, you can list the packages using `PackageReference` without the version component:

```xml
<ItemGroup>
  <PackageReference Include="Newtonsoft.Json" />
  <PackageReference Include="AutoMapper" />
  <PackageReference Include="Polly" />
</ItemGroup>
```

That's it!
Now all your projects will use the same package versions.

## Cool Things You Can Do

### Need a Different Version for One Project?

Sometimes you might need a specific project to use a different version.
No problem!
Just add this to your project file:

```xml
<PackageReference Include="Serilog" VersionOverride="3.1.1" />
```

The `VersionOverride` property lets you define the specific version you want to use.

### Want a Package in Every Project?

If you have packages that every project needs, you can make them global.
Define a `GlobalPackageReference` in your props file:

```xml
<ItemGroup>
  <GlobalPackageReference Include="SonarAnalyzer.CSharp" Version="10.3.0.106239" />
</ItemGroup>
```

Now every project gets this package automatically!

## Migrating Existing Projects to Central Package Management

1. Create the `Directory.Packages.props` file at the solution root
2. Move all package versions from your `.csproj` files
3. Remove version attributes from `PackageReference` elements
4. Build your solution and fix any version conflicts
5. Test thoroughly before committing

Here's a Powershell script that will list all NuGet package versions in your solution:

```powershell
# Scan all .csproj files and aggregate unique package versions
$packages = Get-ChildItem -Filter *.csproj -Recurse |
    Get-Content |
    Select-String -Pattern '<PackageReference Include="([^"]+)" Version="([^"]+)"' -AllMatches |
    ForEach-Object { $_.Matches } |
    Group-Object { $_.Groups[1].Value } |
    ForEach-Object { @{
        Name = $_.Name
        Versions = $_.Group.ForEach({ $_.Groups[2].Value }) | Select-Object -Unique
    }} |
    Sort-Object { $_.Name }

# Display results
$packages | ForEach-Object {
    "$($_.Name) versions:"
    $_.Versions | ForEach-Object { "  $_" }
}
```

There's also a CLI tool called [CentralisedPackageConverter](https://github.com/Webreaper/CentralisedPackageConverter),
which you can use to automate the migration.
It will scan for all .NET project files within that folder tree, gather all the versioned references in the projects,
remove the versions from the project files, and write the entries to the `Directory.Packages.props` file.

```bash
# Install the tool globally
dotnet tool install CentralisedPackageConverter --global

# Convert your solution to use Central Package Management
central-pkg-converter /PATH_TO_YOUR_SOLUTION_FOLDER
```

## When Should You Use CPM?

I don't see a compelling reason for not using this by default.

I recommend using CPM when:

- You have many projects that share packages
- You're tired of fixing version-related bugs
- You want to make sure everyone uses the same versions

I recently added CPM to a solution with 30 projects.

Here's what happened:

- Fewer merge conflicts
- Caught version problems early
- Made it easier for new team members

This was especially helpful while migrating from .NET 8 to .NET 9.

You can combine CPM with [**build configuration and static code analysis**](improving-code-quality-in-csharp-with-static-code-analysis).

## Wrapping Up

My tips for success with **Central Package Management**:

1. When you add CPM to an existing solution, do it in its own change/PR
2. If you override a version, add a comment explaining why
3. Check your package versions regularly for updates
4. Only make packages global if you really need them everywhere

Since I started using Central Package Management, managing NuGet packages has become much easier.
It's like having a single source of truth for all your package versions.

Hope this was helpful.
See you next week.

---



<Sponsorships>

[**Make the Switch: IronPDF vs Aspose**](https://ironpdf.com/competitors/aspose-vs-ironpdf/?utm_source=milan&utm_medium=email&utm_campaign=milan)!
Why overcomplicate your projects?
IronPDF offers premium features like PDF compliance, pixel-perfect rendering, and seamless integration at a fraction of Aspose's cost.
Enjoy one year of free support, straightforward pricing, and a simpler API.
[**Switch & save extra 33%**](https://ironpdf.com/competitors/aspose-vs-ironpdf/?utm_source=milan&utm_medium=email&utm_campaign=milan).

Is API development a straightforward process?
We wish!
No matter, though. [**Postman's November Drop is here to help you stay on track with**](https://shortclick.link/xvvfnr):\
• The Postman VS Code extension.\
• Response-sharing on Slack and Teams.\
• Auto-generating documentation.

</Sponsorships>

Long-running business processes often involve multiple services working together.
Think about an e-commerce order: you need to process the payment, update inventory, and notify shipping.
Traditional distributed transactions using two-phase commit (2PC) seem like a solution, but they come with significant drawbacks.

The main issue?
Services can't make assumptions about how other services operate or how long they'll take.
What if the payment service needs manual approval?
What if the inventory check is delayed?
Holding database locks across multiple services for extended periods isn't practical and can lead to system-wide issues.

Let's look at how the Saga pattern solves these problems and implement it using MassTransit.

## Understanding the Saga Pattern

A Saga is a sequence of related local transactions where each step has a defined action and a compensating action if something goes wrong.
Instead of one big atomic transaction, we break the process into manageable steps that can be coordinated.

Here's a simple order processing flow:

<div className="centered">
  <Image
    src="blogs/mnw_118/order_flow.png"
    width={2313 / 4}
    height={2628 / 4}
    alt="Sequence diagram showing an order processing flow with multiple services."
  />
</div>

Each step is independent and can be compensated if needed.
If the inventory service reports items are out of stock, we can refund the payment.
This approach gives us flexibility and reliability without tight coupling.

## Implementing Sagas with MassTransit

[**MassTransit**](using-masstransit-with-rabbitmq-and-azure-service-bus) provides a state machine-based approach to implementing Sagas through its integration with Automatonymous.
Understanding how state machines work is crucial for implementing effective sagas.

### State Machine Fundamentals

A [state machine](https://masstransit.io/documentation/patterns/saga/state-machine) consists of several key components:

1. **States**: Represent the possible conditions of your saga instance
2. **Events**: Messages that can trigger state transitions
3. **Behaviors**: Actions that occur when events are received in specific states
4. **Instance**: Contains the data and current state for a specific saga

Here's the state machine diagram for our order processing saga:

<div className="centered">
  <Image
    src="blogs/mnw_118/order_saga_state_machine.png"
    width={1633 / 3}
    height={1728 / 3}
    alt="State diagram showing an order saga state machine."
  />
</div>

Every state machine automatically includes `Initial` and `Final` states.
The `Initial` state is where new saga instances begin, and the `Final` state marks the end of a saga's lifecycle.

### Defining the Saga Instance

The saga instance holds the data for a specific process:

```csharp
public class OrderState : SagaStateMachineInstance
{
    public Guid CorrelationId { get; set; }
    public string CurrentState { get; set; }

    // Business data
    public decimal OrderTotal { get; set; }
    public string? PaymentIntentId { get; set; }
    public DateTime? OrderDate { get; set; }
    public string? CustomerEmail { get; set; }
}
```

The `CorrelationId` uniquely identifies the saga instance, while `CurrentState` tracks its current state.
Any additional properties store business data needed for the process.

### Defining Events

Events are messages that can trigger state transitions. They must be correlated to a specific saga instance:

```csharp
public record OrderSubmitted
{
    public Guid OrderId { get; init; }
    public decimal Total { get; init; }
    public string Email { get; init; }
}

public record PaymentProcessed
{
    public Guid OrderId { get; init; }
    public string PaymentIntentId { get; init; }
}

public record InventoryReserved
{
    public Guid OrderId { get; init; }
}

public record OrderFailed
{
    public Guid OrderId { get; init; }
    public string Reason { get; init; }
}
```

### Building the State Machine

Let's implement the order processing flow as a state machine:

```csharp
public class OrderStateMachine : MassTransitStateMachine<OrderState>
{
    public OrderStateMachine()
    {
        Event(() => OrderSubmitted, x => x.CorrelateById(m => m.Message.OrderId));
        Event(() => PaymentProcessed, x => x.CorrelateById(m => m.Message.OrderId));
        Event(() => InventoryReserved, x => x.CorrelateById(m => m.Message.OrderId));
        Event(() => OrderFailed, x => x.CorrelateById(m => m.Message.OrderId));

        InstanceState(x => x.CurrentState);

        Initially(
            When(OrderSubmitted)
                .Then(context =>
                {
                    context.Saga.OrderTotal = context.Message.Total;
                    context.Saga.CustomerEmail = context.Message.Email;
                    context.Saga.OrderDate = DateTime.UtcNow;
                })
                .PublishAsync(context => context.Init<ProcessPayment>(new
                {
                    OrderId = context.Saga.CorrelationId,
                    Amount = context.Saga.OrderTotal
                }))
                .TransitionTo(ProcessingPayment)
        );

        During(ProcessingPayment,
            When(PaymentProcessed)
                .PublishAsync(context => context.Init<ReserveInventory>(new
                {
                    OrderId = context.Saga.CorrelationId
                }))
                .TransitionTo(ReservingInventory),
            When(OrderFailed)
                .TransitionTo(Failed)
                .Finalize()
        );

        During(ReservingInventory,
            When(InventoryReserved)
                .PublishAsync(context => context.Init<OrderConfirmed>(new
                {
                    OrderId = context.Saga.CorrelationId
                }))
                .TransitionTo(Completed)
                .Finalize(),
            When(OrderFailed)
                .PublishAsync(context => context.Init<RefundPayment>(new
                {
                    OrderId = context.Saga.CorrelationId,
                    Amount = context.Saga.OrderTotal
                }))
                .TransitionTo(Failed)
                .Finalize()
        );

        SetCompletedWhenFinalized();
    }

    public State ProcessingPayment { get; private set; }
    public State ReservingInventory { get; private set; }
    public State Completed { get; private set; }
    public State Failed { get; private set; }

    public Event<OrderSubmitted> OrderSubmitted { get; private set; }
    public Event<PaymentProcessed> PaymentProcessed { get; private set; }
    public Event<InventoryReserved> InventoryReserved { get; private set; }
    public Event<OrderFailed> OrderFailed { get; private set; }
}
```

The state machine defines the possible states and transitions.
Each step can trigger compensating actions if needed.
For example, if inventory reservation fails, we automatically trigger a payment refund.

### Implementing Message Consumers

Services interact with the saga by consuming and publishing messages.
Here's an example of a payment processing consumer:

```csharp
public class ProcessPaymentConsumer(
    IPaymentService paymentService,
    ILogger<ProcessPaymentConsumer> logger) : IConsumer<ProcessPayment>
{
    public async Task Consume(ConsumeContext<ProcessPayment> context)
    {
        try
        {
            var paymentResult = await paymentService.ProcessPaymentAsync(
                context.Message.OrderId,
                context.Message.Amount
            );

            if (paymentResult.Succeeded)
            {
                await context.Publish<PaymentProcessed>(new
                {
                    OrderId = context.Message.OrderId,
                    PaymentIntentId = paymentResult.PaymentIntentId
                });
            }
            else
            {
                await context.Publish<OrderFailed>(new
                {
                    OrderId = context.Message.OrderId,
                    Reason = paymentResult.FailureReason
                });
            }
        }
        catch (Exception ex)
        {
            logger.LogError(
                ex,
                "Failed to process payment for order {OrderId}",
                context.Message.OrderId);

            await context.Publish<OrderFailed>(new
            {
                OrderId = context.Message.OrderId,
                Reason = "Payment processing error"
            });
        }
    }
}
```

Each consumer handles a specific part of the business process and communicates back to the saga through events.
This separation of concerns allows each service to focus on its specific responsibility while the saga coordinates the overall process.

## Configuring MassTransit with PostgreSQL Persistence

To persist the saga state, we'll use PostgreSQL.
First, let's install the required packages:

```powershell
Install-Package MassTransit.EntityFrameworkCore
Install-Package Npgsql.EntityFrameworkCore.PostgreSQL
```

Create a `DbContext` for saga persistence:

```csharp
public class OrderSagaDbContext : SagaDbContext
{
    public OrderSagaDbContext(DbContextOptions options) : base(options)
    {
    }

    protected override IEnumerable<ISagaClassMap> Configurations
    {
        get
        {
            yield return new OrderStateMap();
        }
    }
}

public class OrderStateMap : SagaClassMap<OrderState>
{
    protected override void Configure(EntityTypeBuilder<OrderState> entity, ModelBuilder model)
    {
        entity.Property(x => x.CurrentState).HasMaxLength(64);
        entity.Property(x => x.CustomerEmail).HasMaxLength(256);
        entity.Property(x => x.PaymentIntentId).HasMaxLength(64);
    }
}
```

Configure MassTransit in your application:

```csharp
builder.Services.AddDbContext<OrderSagaDbContext>(options =>
    options.UseNpgsql(builder.Configuration.GetConnectionString("Postgres")));

builder.Services.AddMassTransit(x =>
{
    x.AddSagaStateMachine<OrderStateMachine, OrderState>()
        .EntityFrameworkRepository(r =>
        {
            r.ConcurrencyMode = ConcurrencyMode.Pessimistic;
            r.AddDbContext<DbContext, OrderSagaDbContext>();
            r.UsePostgres();
        });

    x.UsingRabbitMq((context, cfg) =>
    {
        cfg.Host(builder.Configuration.GetConnectionString("RabbitMQ"));
        cfg.ConfigureEndpoints(context);
    });
});
```

## Benefits of This Approach

Using sagas with MassTransit provides several advantages:

1. **Fault Tolerance**: Each step can be retried independently.
   And we can compensate failed operation, making our system more resilient.
2. **State Visibility**: The saga's state machine provides clear insight into where each process stands,
   making debugging and monitoring straightforward.
3. **Loose Coupling**: Services communicate through messages, allowing them to evolve independently while maintaining process integrity.
4. **Maintainability**: Changes can be made to individual steps without affecting others.

The state machine approach also makes the business process explicit.
Each state and transition is clearly defined, making it easier to understand and maintain the workflow.

## Takeaway

The Saga pattern with MassTransit provides a robust solution for managing distributed business processes.
Instead of dealing with distributed transactions, you get clear state management, automatic compensation for failures,
and the ability to handle long-running operations without blocking resources.

Want to dive deeper into event-driven architecture and distributed systems?
Check out my [**Modular Monolith Architecture**](/modular-monolith-architecture) course,
where we explore sagas, event-driven patterns, and other essential techniques for building maintainable systems.

Good luck out there, and see you next week.

**P.S.** You can find the code for the saga pattern implementation with MassTransit in [this repository](https://github.com/m-jovanovic/saga-pattern-masstransit).

---



<Sponsorships>

Even the smartest AI in the world won't save you from a productivity doom spiral if you don't get your APIs in order.
Don't spiral.
Do this instead — [**Check out Postman's November Drop to learn how the Postman VS Code extension can help**](https://shortclick.link/xvvfnr).
Avoid switching back and forth between tools.
Share responses on Slack and Teams.
Auto-generate your documentation.

[**Multiplayer**](https://dub.sh/e1TYNOD) auto-documents your system, from the high-level logical architecture down to the individual components, APIs, and dependencies.
Perfect for teams who want to speed up their workflows and consolidate their technical assets.
[**Try it for free here**](https://dub.sh/e1TYNOD).

</Sponsorships>

Most APIs follow a simple pattern.
The client sends a request.
The server does some work.
The server sends back a response.

This works well for fast operations like fetching data or simple updates.
But what about operations that take longer?

Think about processing large files, generating reports, or converting videos.
These operations can take minutes or even hours.

Making clients wait for these operations causes problems.

## Understanding Async APIs

The key to handling long-running operations is to change how we think about API responses.
An async API splits work into two parts:

- Accept the request
- Process it later

First, we accept the request and return a tracking ID immediately.
This gives users a quick response.
Then, we process the actual work in the background, which won't block other requests.
Users can check the status of their request using the tracking ID whenever they want.

This is different from `async`/`await` in C#.
That's about handling many requests at once (concurrently).
This is about handling long-running tasks better.
We're not just making the code asynchronous - we're making the entire operation asynchronous from the user's perspective.

## The Problem with Sync APIs

Let's see this in practice with image processing. A typical image upload API might look like this:

```csharp
[HttpPost]
public async Task<IActionResult> UploadImage(IFormFile file)
{
    if (file is null)
    {
        return BadRequest();
    }

    // Save original image
    var originalPath = await SaveOriginalAsync(file);

    // Generate thumbnails
    var thumbnails = await GenerateThumbnailsAsync(originalPath);

    // Optimize all images
    await OptimizeImagesAsync(originalPath, thumbnails);

    return Ok(new { originalPath, thumbnails });
}
```

The client must wait while we save the file, generate thumbnails, and optimize images.
On a slow connection or with a large file, this request could time out.
The server is also stuck processing one image at a time.

<Image
  src="blogs/mnw_117/sync_api_request.png"
  width={2460}
  height={1632}
  alt="Sequence diagram showing a synchronous API request."
/>

## A Better Way: Async Processing

Let's fix these problems. We'll split the work into two parts:

1. Accept the upload and return quickly
2. Do the heavy work in the background

<Image
  src="blogs/mnw_117/async_api_request.png"
  width={3275}
  height={2715}
  alt="Sequence diagram showing an asynchronous API request."
/>

### Uploading Images

Here's the new upload endpoint:

```csharp
[HttpPost]
public async Task<IActionResult> UploadImage(IFormFile? file)
{
    if (file is null)
    {
        return BadRequest("No file uploaded.");
    }

    if (!imageService.IsValidImage(file))
    {
        return BadRequest("Invalid image file.");
    }

    // Phase 1: Accept the work
    var id = Guid.NewGuid().ToString();
    var folderPath = Path.Combine(_uploadDirectory, "images", id);
    var fileName = $"{id}{Path.GetExtension(file.FileName)}";
    var originalPath = await imageService.SaveOriginalImageAsync(
        file,
        folderPath,
        fileName
    );

    // Queue Phase 2 for background processing
    var job = new ImageProcessingJob(id, originalPath, folderPath);
    await jobQueue.EnqueueAsync(job);

    // Return status URL immediately
    var statusUrl = GetStatusUrl(id);
    return Accepted(statusUrl, new { id, status = "queued" });
}
```

This new version only saves the original file during the HTTP request.
The heavy work moves to a background process.
The client immediately gets a status URL in the `Location` header instead of waiting.

### Checking Progress

Clients can check their image's status using a separate endpoint:

```csharp
[HttpGet("{id}/status")]
public IActionResult GetStatus(string id)
{
    if (!statusTracker.TryGetStatus(id, out var status))
    {
        return NotFound();
    }

    var response = new
    {
        id,
        status,
        links = new Dictionary<string, string>()
    };

    if (status == "completed")
    {
        response.links = new Dictionary<string, string>
        {
            ["original"] = GetImageUrl(id),
            ["thumbnail"] = GetThumbnailUrl(id, width: 200),
            ["preview"] = GetThumbnailUrl(id, width: 800)
        };
    }

    return Ok(response);
}
```

### Processing Images in Background

The real work happens in the background processor.
While the API handles new requests, a separate process works through the queued jobs.
This separation gives us flexibility in how we handle the processing.

For single-server deployments, we can use .NET's [**Channel**](lightweight-in-memory-message-bus-using-dotnet-channels) type to queue jobs in memory:

```csharp
public class JobQueue
{
    private readonly Channel<ImageProcessingJob> _channel;

    public JobQueue()
    {
        var options = new BoundedChannelOptions(1000)
        {
            FullMode = BoundedChannelFullMode.Wait
        };
        _channel = Channel.CreateBounded<ImageProcessingJob>(options);
    }

    public async ValueTask EnqueueAsync(ImageProcessingJob job,
        CancellationToken ct = default)
    {
        await _channel.Writer.WriteAsync(job, ct);
    }

    public IAsyncEnumerable<ImageProcessingJob> DequeueAsync(
        CancellationToken ct = default)
    {
        return _channel.Reader.ReadAllAsync(ct);
    }
}
```

For multi-server setups, we need a distributed queue like RabbitMQ or even Redis.

The background processor handles the time-consuming work:

```csharp
public class ImageProcessor : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        await foreach (var job in jobQueue.DequeueAsync(ct))
        {
            try
            {
                await statusTracker.SetStatusAsync(
                    job.Id,
                    "processing"
                );

                // Generate thumbnails
                await GenerateThumbnailsAsync(
                    job.OriginalPath,
                    job.OutputPath
                );

                // Optimize images
                await OptimizeImagesAsync(
                    job.OriginalPath,
                    job.OutputPath
                );

                await statusTracker.SetStatusAsync(
                    job.Id,
                    "completed"
                );
            }
            catch (Exception ex)
            {
                await statusTracker.SetStatusAsync(
                    job.Id,
                    "failed"
                );

                logger.LogError(ex, "Failed to process image {Id}", job.Id);
            }
        }
    }
}
```

The background processor needs to handle failures gracefully.
We can improve [**resilience**](building-resilient-cloud-applications-with-dotnet) by adding a retry policy with Polly.
Status updates keep users informed throughout the process.
Instead of just "processing", we tell them exactly what's happening.
This improves the user experience and helps with debugging.

## Beyond Polling: Real-Time Updates

Our status endpoint works, but it puts the burden on clients.
They must repeatedly check for updates, leading to unnecessary server load.
A client polling every second creates 60 requests per minute, yet most of these requests return the same status.

We can flip this model around.
Instead of clients asking for updates, the server can push updates when they happen.
This creates a more efficient and responsive system.

<Image
  src="blogs/mnw_117/async_api_request_with_push.png"
  width={3732}
  height={2964}
  alt="Sequence diagram showing an asynchronous API request with server push for status updates."
/>

[**SignalR and WebSockets**](adding-real-time-functionality-to-dotnet-applications-with-signalr) enable real-time communication between server and client.
When a job's status changes, the server immediately notifies interested clients.
This approach reduces network traffic and gives users instant feedback.

For longer-running jobs, email notifications make more sense.
Users don't need to keep their browsers open.
They can close the tab and come back when notified.
This works well for reports that take hours to generate or batch processes that run overnight.

Webhooks offer another option, especially for system-to-system communication.
When a job completes, your server can notify other systems.
This enables workflow automation and system integration without constant polling.

## Summary

Processing tasks asynchronously creates better experiences for everyone.
Users get immediate responses instead of watching spinning loading indicators.
They can start other tasks while waiting, and they'll know if something goes wrong.

The benefits extend beyond user experience.
Servers can handle more requests because they're not tied up with long-running tasks.
[**Background processors**](scheduling-background-jobs-with-quartz-net) can retry failed operations without affecting the main application.
You can even scale your processing separately from your web servers.

[**Error handling**](global-error-handling-in-aspnetcore-8) improves too.
When a long operation fails halfway through, you can save the progress and try again.
Users know exactly what's happening because they can check the status.
The system stays stable because one slow operation can't bring down your entire API.

That's all for today. Hope this was helpful.

---



<Sponsorships>

🎉 [**JetBrains Rider**](https://www.jetbrains.com/rider/?utm_campaign=rider_free&utm_content=site&utm_medium=cpc&utm_source=milan_jovanovic_newsletter)
is now free for non-commercial use.
Great news for all hobbyists, students, content creators, and open source contributors!
Now you can use Rider, a cross-platform .NET and game dev IDE, for non-commercial development for free.
[**Download and start today**](https://www.jetbrains.com/rider/?utm_campaign=rider_free&utm_content=site&utm_medium=cpc&utm_source=milan_jovanovic_newsletter)!

Remember when APIs were an afterthought?
We're [**not going back**](https://shortclick.link/djx5wv).
74% of devs surveyed in the 2024 State of the API Report are API-first, up from 66% in 2023.
What else? We're getting faster. 63% of devs can produce an API within a week, up from 47% in 2023.
Collaboration on APIs is still a challenge.
APIs are increasingly revenue generators.
Read the [**rest of the report**](https://shortclick.link/djx5wv) for fresh insights.

</Sponsorships>

[**Caching**](caching-in-aspnetcore-improving-application-performance) is essential for building fast, scalable applications.
ASP.NET Core has traditionally offered two caching options: in-memory caching and distributed caching.
Each has its trade-offs.
In-memory caching using `IMemoryCache` is fast but limited to a single server.
Distributed caching with `IDistributedCache` works across multiple servers using a backplane.

.NET 9 introduces `HybridCache`, a new library that combines the best of both approaches.
It prevents common caching problems like cache stampede.
It also adds useful features like tag-based invalidation and better performance monitoring.

In this week's issue, I'll show you how to use `HybridCache` in your applications.

## What is HybridCache?

The traditional caching options in ASP.NET Core have limitations.
In-memory caching is fast but limited to one server.
Distributed caching works across servers but is slower.

[HybridCache](https://learn.microsoft.com/en-us/aspnet/core/performance/caching/hybrid) combines both approaches and adds important features:

- Two-level caching (L1/L2)
  - L1: Fast in-memory cache
  - L2: Distributed cache (Redis, SQL Server, etc.)
- Protection against [cache stampede](https://en.wikipedia.org/wiki/Cache_stampede) (when many requests hit an empty cache at once)
- Tag-based cache invalidation
- Configurable serialization
- Metrics and monitoring

The L1 cache runs in your application's memory.
The L2 cache can be Redis, SQL Server, or any other distributed cache.
You can use HybridCache with just the L1 cache if you don't need distributed caching.

## Installing HybridCache

Install the `Microsoft.Extensions.Caching.Hybrid` NuGet package:

```powershell
Install-Package Microsoft.Extensions.Caching.Hybrid
```

Add `HybridCache` to your services:

```csharp
builder.Services.AddHybridCache(options =>
{
    // Maximum size of cached items
    options.MaximumPayloadBytes = 1024 * 1024 * 10; // 10MB
    options.MaximumKeyLength = 512;

    // Default timeouts
    options.DefaultEntryOptions = new HybridCacheEntryOptions
    {
        Expiration = TimeSpan.FromMinutes(30),
        LocalCacheExpiration = TimeSpan.FromMinutes(30)
    };
});
```

For custom types, you can add your own serializer:

```csharp
builder.Services.AddHybridCache()
    .AddSerializer<CustomType, CustomSerializer>();
```

## Using HybridCache

`HybridCache` provides several methods to work with cached data.
The most important ones are `GetOrCreateAsync`, `SetAsync`, and various remove methods.
Let's see how to use each one in real-world scenarios.

### Getting or Creating Cache Entries

The `GetOrCreateAsync` method is your main tool for working with cached data.
It handles both cache hits and misses automatically.
If the data isn't in the cache, it calls your factory method to get the data, caches it, and returns it.

Here's an endpoint that gets product details:

```csharp
app.MapGet("/products/{id}", async (
    int id,
    HybridCache cache,
    ProductDbContext db,
    CancellationToken ct) =>
{
    var product = await cache.GetOrCreateAsync(
        $"product-{id}",
        async token =>
        {
            return await db.Products
                .Include(p => p.Category)
                .FirstOrDefaultAsync(p => p.Id == id, token);
        },
        cancellationToken: ct
    );

    return product is null ? Results.NotFound() : Results.Ok(product);
});
```

In this example:

- The cache key is unique per product
- If the product is in the cache, it's returned immediately
- If not, the factory method runs to get the data
- Other concurrent requests for the same product wait for the first one to finish

### Setting Cache Entries Directly

Sometimes you need to update the cache directly, like after modifying data.
The `SetAsync` method handles this:

```csharp
app.MapPut("/products/{id}", async (int id, Product product, HybridCache cache) =>
{
    // First update the database
    await UpdateProductInDatabase(product);

    // Then update the cache with custom expiration
    var options = new HybridCacheEntryOptions
    {
        Expiration = TimeSpan.FromHours(1),
        LocalCacheExpiration = TimeSpan.FromMinutes(30)
    };

    await cache.SetAsync(
        $"product-{id}",
        product,
        options
    );

    return Results.NoContent();
});
```

Key points about `SetAsync`:

- It updates both L1 and L2 cache
- You can specify different timeouts for L1 and L2
- It overwrites any existing value for the same key

### Using Cache Tags

Tags are powerful for managing groups of related cache entries.
You can invalidate multiple entries at once using tags:

```csharp {7,18,31}
app.MapGet("/categories/{id}/products", async (
    int id,
    HybridCache cache,
    ProductDbContext db,
    CancellationToken ct) =>
{
    var tags = [$"category-{id}", "products"];

    var products = await cache.GetOrCreateAsync(
        $"products-by-category-{id}",
        async token =>
        {
            return await db.Products
                .Where(p => p.CategoryId == id)
                .Include(p => p.Category)
                .ToListAsync(token);
        },
        tags: tags,
        cancellationToken: ct
    );

    return Results.Ok(products);
});

// Endpoint to invalidate all products in a category
app.MapPost("/categories/{id}/invalidate", async (
    int id,
    HybridCache cache,
    CancellationToken ct) =>
{
    await cache.RemoveByTagAsync($"category-{id}", ct);

    return Results.NoContent();
});
```

Tags are useful for:

- Invalidating all products in a category
- Clearing all cached data for a specific user
- Refreshing all related data when something changes

### Removing Single Entries

For direct cache invalidation of specific items, use `RemoveAsync`:

```csharp
app.MapDelete("/products/{id}", async (int id, HybridCache cache) =>
{
    // First delete from database
    await DeleteProductFromDatabase(id);

    // Then remove from cache
    await cache.RemoveAsync($"product-{id}");

    return Results.NoContent();
});
```

`RemoveAsync`:

- Removes the item from both L1 and L2 cache
- Works immediately, no delay
- Does nothing if the key doesn't exist
- Is safe to call multiple times

Remember that `HybridCache` handles all the complexity of distributed caching, serialization, and stampede protection for you.
You just need to focus on your cache keys and when to invalidate the cache.

## Adding Redis as L2 Cache

To use [Redis](https://redis.io/) as your distributed cache:

1. Install the `Microsoft.Extensions.Caching.StackExchangeRedis` NuGet package:

```powershell
Install-Package Microsoft.Extensions.Caching.StackExchangeRedis
```

2. Configure Redis and `HybridCache`:

```csharp
// Add Redis
builder.Services.AddStackExchangeRedisCache(options =>
{
    options.Configuration = "your-redis-connection-string";
});

// Add HybridCache - it will automatically use Redis as L2
builder.Services.AddHybridCache();
```

`HybridCache` will automatically detect and use Redis as the L2 cache.

## Summary

`HybridCache` simplifies caching in .NET applications.
It combines fast in-memory caching with distributed caching, prevents common problems like cache stampede,
and works well in both single-server and distributed systems.

Start with the default settings and basic usage patterns - the library is designed to be simple to use while solving complex caching problems.

Thanks for reading.

And stay awesome!

---


Previous issues of The .NET Weekly