
<Sponsorships>

Visually Building API-Driven Applications with [**Postman Flows**](https://app.zuddl.com/p/a/event/d2d123ee-fb4c-4359-9b38-cb471fe1a844?utm_source=influencer&utm_medium=referral&utm_campaign=fy25q3_global_ent_postman_flows_webinar&utm_term=existing_influencer_relationships&utm_content=milan_jovanovic).
Explore how Postman Flows demystifies the composition of API applications API workflows through a low-code experience that lets you visualize complex API workflows.
[**Register here**](https://app.zuddl.com/p/a/event/d2d123ee-fb4c-4359-9b38-cb471fe1a844?utm_source=influencer&utm_medium=referral&utm_campaign=fy25q3_global_ent_postman_flows_webinar&utm_term=existing_influencer_relationships&utm_content=milan_jovanovic).

Deliver a great UI with easily customizable [**Page Templates and Building Blocks**](https://www.telerik.com/page-templates-and-ui-blocks?utm_medium=cpm&utm_source=milanjonanovich&utm_campaign=dt_design_templates_building_blocks).
Utilize a collection of over 50 building blocks and 10 page templates designed to help you build modern UI in no time.
All you need to do is copy and paste!
[**Learn more here**](https://www.telerik.com/page-templates-and-ui-blocks?utm_medium=cpm&utm_source=milanjonanovich&utm_campaign=dt_design_templates_building_blocks).

</Sponsorships>

In distributed systems, we often face the challenge of keeping our database and external systems in sync.
Imagine saving an order to a database and then publishing a message to a message broker.
If either operation fails, your system ends up in an inconsistent state.

The [**Outbox pattern**](outbox-pattern-for-reliable-microservices-messaging) solves this problem by treating message publication as part of your database transaction.
Instead of publishing messages directly, we save them to an Outbox table in our database, ensuring atomic operations.
A separate process then reliably publishes these messages.

In this newsletter, we'll dive into implementing this pattern in .NET, covering everything from setup to scaling.

## Why Do We Need the Outbox Pattern?

The transactional Outbox pattern fixes a common problem in distributed systems.
This problem happens when you need to do two things at once: save data and communicate with an external component.

Consider scenarios like sending order confirmation emails, notifying other systems about new client registrations,
or updating inventory levels after an order is placed.
Each of these involves a local data change coupled with an external communication or update.

For example, imagine a microservice that needs to:

- Save a new order in its database
- Tell other systems about this new order

If one of these steps fails, your system could end up in an inconsistent state.
Maybe the order is saved, but no one else knows about it.
Or everyone thinks there's a new order, but it's not actually in the database.

Here's a `CreateOrderCommandHandler` without the Outbox pattern:

```csharp
public class CreateOrderCommandHandler(
    IOrderRepository orderRepository,
    IProductInventoryChecker inventoryChecker,
    IUnitOfWork unitOfWork,
    IEventBus eventBus) : IRequestHandler<CreateOrderCommand, OrderDto>
{
    public async Task<OrderDto> Handle(CreateOrderCommand request, CancellationToken cancellationToken)
    {
        var order = new Order(request.CustomerId, request.ProductId, request.Quantity, inventoryChecker);

        await orderRepository.AddAsync(order);

        await unitOfWork.CommitAsync(cancellationToken);

        // The database transaction is completed at this point.

        await eventBus.Send(new OrderCreatedIntegrationEvent(order.Id));

        return new OrderDto { Id = order.Id, Total = order.Total };
    }
}
```

This code has a potential consistency problem.
After the database transaction is committed, two things could go wrong:

1. The application might crash right after the transaction is committed but before the event is sent.
   The order would be created in the database, but other systems wouldn't know about it.

2. The event bus might be down or unreachable when we try to send the event.
   This would result in the order being created without notifying other systems.

The transactional Outbox pattern helps solve this problem by ensuring that the database update and event publication are treated as a single atomic operation.

<Image
  src="blogs/mnw_110/outbox_pattern.png"
  width={2102}
  height={1372}
  alt="Flow diagram explaining how the Outbox Pattern works."
/>

The sequence diagram illustrates how the Outbox pattern solves our consistency challenge.
Instead of trying to save data and send a message as separate steps, we save both the order and an Outbox message in a **single database transaction**.
This is an all-or-nothing operation - we can't end up in an inconsistent state.

A separate Outbox processor handles the actual message sending.
It continuously checks for unsent messages in the Outbox table and publishes them to the message queue.
The processor marks messages as sent after successful publishing, preventing duplicates.

An important thing to realize here is that the Outbox pattern gives us [at-least-once delivery](https://www.cloudcomputingpatterns.org/at_least_once_delivery/).
The Outbox message will be sent at least once, but it could also be sent multiple times in case of retries.
This means we have to make our message consumers idempotent.

## Implementing the Outbox Pattern

First, let's create our Outbox table where we will store messages:

```sql
CREATE TABLE outbox_messages (
    id UUID PRIMARY KEY,
    type VARCHAR(255) NOT NULL,
    content JSONB NOT NULL,
    occurred_on_utc TIMESTAMP WITH TIME ZONE NOT NULL,
    processed_on_utc TIMESTAMP WITH TIME ZONE NULL,
    error TEXT NULL
);

-- We can consider adding this index since we will be querying for unprocessed messages often
-- and it will contain the rows in the correct sort order for our query.
CREATE INDEX IF NOT EXISTS idx_outbox_messages_unprocessed
ON outbox_messages (occurred_on_utc, processed_on_utc)
INCLUDE (id, type, content)
WHERE processed_on_utc IS NULL;
```

I'll use PostgreSQL as the database for this example.
Notice the `jsonb` type for the `content` column.
It allows for indexing and querying of the JSON data if needed in the future.

Now, let's create a class to represent our Outbox entry:

```csharp
public sealed class OutboxMessage
{
    public Guid Id { get; init; }
    public string Type { get; init; }
    public string Content { get; init; }
    public DateTime OccurredOnUtc { get; init; }
    public DateTime? ProcessedOnUtc { get; init; }
    public string? Error { get; init; }
}
```

Here's how we can add a message to the Outbox:

```csharp
public async Task AddToOutbox<T>(T message, NpgsqlDataSource dataSource)
{
    var outboxMessage = new OutboxMessage
    {
        Id = Guid.NewGuid(),
        OccurredOnUtc = DateTime.UtcNow,
        Type = typeof(T).FullName, // We'll need this for deserialization
        Content = JsonSerializer.Serialize(message)
    };

    await using var connection = await dataSource.OpenConnectionAsync();
    await connection.ExecuteAsync(
        @"""
        INSERT INTO outbox_messages (id, occurred_on_utc, type, content)
        VALUES (@Id, @OccurredOnUtc, @Type, @Content::jsonb)
        """,
        outboxMessage);
}
```

An elegant approach to implementing this is using [**domain events**](how-to-use-domain-events-to-build-loosely-coupled-systems) to represent notifications.
When something significant happens in the domain, we will raise a domain event.
Before completing the transaction, we can pick up all events and store them as Outbox messages.
You could do this from the unit of work or with an [**EF Core interceptor**](how-to-use-ef-core-interceptors).

## Processing the Outbox

The Outbox processor is the next component we'll need.
This could be a _physically_ separate process or a background worker in the same process.

I'll use Quartz to [**schedule background jobs**](scheduling-background-jobs-with-quartz-net) for Outbox processing.
It's a robust library with excellent support for scheduling recurring jobs.

Now, let's implement the `OutboxProcessorJob`:

```csharp
[DisallowConcurrentExecution]
public class OutboxProcessorJob(
    NpgsqlDataSource dataSource,
    IPublishEndpoint publishEndpoint,
    Assembly integrationEventsAssembly) : IJob
{
    public async Task Execute(IJobExecutionContext context)
    {
        await using var connection = await dataSource.OpenConnectionAsync();
        await using var transaction = await connection.BeginTransactionAsync();

        // You can make the limit a parameter, to control the batch size.
        // We can also select just the id, type, and content columns.
        var messages = await connection.QueryAsync<OutboxMessage>(
            @"""
            SELECT id AS Id, type AS Type, content AS Content
            FROM outbox_messages
            WHERE processed_on_utc IS NULL
            ORDER BY occurred_on_utc LIMIT 100
            """,
            transaction: transaction);

        foreach (var message in messages)
        {
            try
            {
                var messageType = integrationEventsAssembly.GetType(message.Type);
                var deserializedMessage = JsonSerializer.Deserialize(message.Content, messageType);

                // We should introduce retries here to improve reliability.
                await publishEndpoint.Publish(deserializedMessage);

                await connection.ExecuteAsync(
                    @"""
                    UPDATE outbox_messages
                    SET processed_on_utc = @ProcessedOnUtc
                    WHERE id = @Id
                    """,
                    new { ProcessedOnUtc = DateTime.UtcNow, message.Id },
                    transaction: transaction);
            }
            catch (Exception ex)
            {
                // We can also introduce error logging here.

                await connection.ExecuteAsync(
                    @"""
                    UPDATE outbox_messages
                    SET processed_on_utc = @ProcessedOnUtc, error = @Error
                    WHERE id = @Id
                    """,
                    new { ProcessedOnUtc = DateTime.UtcNow, Error = ex.ToString(), message.Id },
                    transaction: transaction);
            }
        }

        await transaction.CommitAsync();
    }
}
```

This approach uses polling to periodically fetch unprocessed messages from the database.
Polling can increase the load on the database, as we'll need to query for unprocessed messages frequently.

An alternative way to process Outbox messages is by using [Transaction log tailing](https://microservices.io/patterns/data/transaction-log-tailing.html).
We can implement this using [Postgres logical replication](https://www.npgsql.org/doc/replication.html).
The database will stream changes from the Write-Ahead Log (WAL) to our application, and we'll process these messages and publish them to the message broker.
You can use this to implement a push-based Outbox processor.

## Considerations and Tradeoffs

The Outbox pattern, while effective, introduces additional complexity and database writes.
In high-throughput systems, it's crucial to monitor its performance to ensure it doesn't become a bottleneck.

I recommend implementing retry mechanisms in the Outbox processor to [**improve reliability**](building-resilient-cloud-applications-with-dotnet).
Consider using exponential backoff for transient failures and a circuit breaker for persistent issues to prevent system overload during outages.

It's essential that you implement [**idempotent message consumers**](idempotent-consumer-handling-duplicate-messages).
Network issues or processor restarts can lead to multiple deliveries of the same message, so your consumers must handle repeated processing safely.

Over time, the Outbox table can grow significantly, potentially impacting database performance.
It's important to implement an archiving strategy early on.
Consider moving processed messages to cold storage or deleting them after a set period.

## Scaling Outbox Processing

As your system grows, you may find that a single Outbox processor can't keep up with the volume of messages.
This can lead to increased latency between when an event occurs and when it's processed by consumers.

One straightforward approach is to increase the frequency of the Outbox processor job.
You should consider running it every few seconds.
This can significantly reduce the delay in message processing.

Another effective strategy is to increase the batch size when fetching unprocessed messages.
By processing more messages in each run, you can improve throughput.
However, be cautious not to make the batches so large that they cause long-running transactions.

For high-volume systems, processing the Outbox in parallel can be very effective.
Implement a locking mechanism to claim batches of messages, allowing multiple processors to work simultaneously without conflict.
You can use `SELECT ... FOR UPDATE SKIP LOCKED` to claim a batch of messages.
This approach can dramatically increase your processing capacity.

## Wrapping Up

The Outbox pattern is a powerful tool for maintaining data consistency in distributed systems.
By decoupling database operations from message publishing, the Outbox pattern ensures that your system remains reliable even in the face of failures.

Remember to keep your consumers idempotent, implement proper scaling strategies, and manage your Outbox table growth.

While it adds some complexity, the benefits of guaranteed message delivery make it a valuable pattern in many scenarios.

If you're looking to implement the Outbox pattern in a robust, production-ready way, you can check out [**Pragmatic Clean Architecture**](/pragmatic-clean-architecture).
It includes an entire section on implementing the Outbox pattern, along with other essential patterns for building maintainable and scalable .NET applications.

That's all for today. Stay awesome, and I'll see you next week.

---



<Sponsorships>

Integrate e-signatures into your workflows with the [**BoldSign .NET SDK**](https://boldsign.com/esignature-api/?utm_source=milanjovanovic&utm_medium=newsletter_ad&utm_campaign=boldsign).
The e-signature API lets you embed signature requests, verify signer identity, and add custom branding.
Free coding assistance and migration support is available.
Try the [**free API sandbox**](https://boldsign.com/esignature-api/?utm_source=milanjovanovic&utm_medium=newsletter_ad&utm_campaign=boldsign) today!

[**Webinar: "Unlocking AI's Potential: Why the Future Depends on APIs"**](https://app.zuddl.com/p/a/event/8e8f96c1-99d5-4392-96a1-c68b8c8b9d2d?utm_source=influencer&utm_medium=referral&utm_campaign=fy25q3_global_ent_unlocking_ai_potential_webinar&utm_term=existing_influencer_relationships&utm_content=milan_jovanovic).
Join Postman CTO, Ankit Sobti, and Head of Customer Experience and Success, Kristine Chin,
at this free webinar to maximize the success of your API products
and provide a world-class experience for your developers, partners, and customers.
[**Join here**](https://app.zuddl.com/p/a/event/8e8f96c1-99d5-4392-96a1-c68b8c8b9d2d?utm_source=influencer&utm_medium=referral&utm_campaign=fy25q3_global_ent_unlocking_ai_potential_webinar&utm_term=existing_influencer_relationships&utm_content=milan_jovanovic).

</Sponsorships>

As your application grows, you might find yourself considering a move from a modular monolith to microservices.
This transition isn't just a technical shift.
It's a strategic move that can reshape how your entire system operates.
But let's be clear: it's not a magic solution and comes with its own challenges.

In this article, I'll share my experience of migrating from a modular monolith to microservices.

We'll explore why you might consider microservices, how to prepare for the migration and the critical steps in the migration process -
from choosing your first module to implementing inter-service communication and managing data migration.

Let's dive in!

## Why Consider Microservices?

A [**modular monolith**](what-is-a-modular-monolith) serves as an excellent starting point for many applications.
It's simpler to develop, easier to understand, and faster to deploy than a distributed system.
But as your application grows, you might start facing some problems.

Here are the key challenges I've encountered with modular monoliths:

- High-load modules can become bottlenecks, affecting the entire system's performance
- As the monolith grows, deployments become riskier and more time-consuming
- Large teams working on a single codebase often step on each other's toes
- You're locked into a single technology stack for the entire application

Microservices can address these issues, but it's not a decision to be taken lightly.

<Image
  src="blogs/mnw_109/modular_monolith_vs_microservices.png"
  width={1646}
  height={685}
  alt="Modular monolith to microservices comparison."
/>

Here's how microservices can help:

- **Scalability**: You can scale individual services based on their specific load, optimizing resource use.
- **Independent deployability**: Updates to one service don't require redeploying the entire application, reducing risk and downtime.
- **Team autonomy**: Separate teams can own different services, leading to faster development cycles.

The good news? If you've built a well-designed modular monolith, you're already halfway there.
The clear boundaries between modules in your monolith can serve as a blueprint for your microservices architecture.

Remember, microservices come with their own complexities in areas like data consistency and inter-service communication.
But for the right use cases, they can provide the flexibility and scalability needed for growing applications.

## Preparing for Migration

The success of your migration largely depends on how well you prepare.
Your best starting point is a well-structured modular monolith.

Here are key areas to focus on:

1. **Review module boundaries**:
   I can't stress this enough: clear module boundaries are crucial.
   Each module should have a distinct responsibility and minimal dependencies on others.
   If boundaries are blurry, refactor before migrating.

2. **Ensure proper data encapsulation**:
   Modules should not directly access each other's data stores.
   I make sure each module owns and manages its data exclusively.
   No shared tables, no direct database access across modules.
   This clean separation makes it much easier to extract modules into microservices later.

3. **Implement clean public APIs**:
   Define clear contracts between modules to facilitate future separation.
   Modules should communicate through well-defined APIs, not by accessing each other's internals.

Here's an example of the above:

```csharp {7,20}
// Good: Clear module boundary
namespace BookingsModule;

public class CreateBooking
{
    private readonly IBookingRepository _bookingRepository;
    private readonly IPaymentGateway _paymentGateway; // Public API

    public CreateBooking(IBookingRepository bookingRepository, IPaymentGateway paymentGateway)
    {
        _bookingRepository = bookingRepository;
        _paymentGateway = paymentGateway;
    }

    public async Task<BookingResult> CreateBookingAsync(BookingRequest request)
    {
        var booking = await _bookingRepository.CreateAsync(Booking.FromRequest(request));

        // Accessing the other module's data through an abstraction
        var paymentResult = await _paymentGateway.ProcessPaymentAsync(booking.Id, booking.TotalAmount);

        return new BookingResult(booking, paymentResult);
    }
}
```

By focusing on these areas, you're not just preparing for migration - you're improving your modular monolith.
Even if you decide not to migrate, these steps will make your system more maintainable and scalable.

## Choosing and Extracting the First Module

Selecting the right module to extract first can set the tone for your entire migration.
In my experience, it's crucial to start with a module that's self-contained and has clear boundaries.

When I'm evaluating modules, I look for these characteristics:

- Low coupling with other modules
- High cohesion within the module
- A distinct business function
- Potential performance or scalability gains from separation

For instance, in an e-commerce system, I might choose the product catalog module.
It typically has a clear purpose, doesn't heavily depend on other modules, and could benefit from independent scaling.

Once you've chosen your module, here's the extraction process I follow:

1. Create a new project for the microservice.
2. Move the module's code to the new project.
   This often reveals hidden dependencies, which is valuable information.
3. Update the dependencies, ensuring the microservice is self-contained.
   This might involve copying some shared code or refactoring to remove unnecessary dependencies.
4. Set up a separate database for the microservice.
   This enforces data independence.
5. Implement a data migration strategy (more on this later)

Here's what this might look like in practice:

<Image
  src="blogs/mnw_109/modular_monolith_extraction.png"
  width={1088}
  height={1184}
  alt="Modular monolith extraction process into microservices."
/>

Remember, the goal isn't perfection on the first try.
I often iterate on this process, gradually refining the separation between the new microservice and the monolith.

The first extraction is a learning experience.
It'll show you how to approach subsequent modules and help you refine the overall migration strategy.

## Implementing Inter-Service Communication

Once you've extracted a module into a microservice, you have to update the [**module's communication**](modular-monolith-communication-patterns)
with the rest of the system.
This typically involves transitioning from direct method calls to network-based communication.

Here's how I approach this transition:

1. Replace direct method calls with HTTP API calls.
   I often use libraries like [**Refit**](refit-in-dotnet-building-robust-api-clients-in-csharp) to simplify API interactions.
   Here's a before and after example:

```csharp
// Before: Direct method call
BookingDto booking = await _bookingService.GetAsync(bookingId);

// After: HTTP API call
var response = await _httpClient.GetAsync($"http://bookings-service/api/bookings/{bookingId}");

var booking = await response.Content.ReadFromJsonAsync<BookingDto>();
```

2. For asynchronous communication, consider implementing a messaging system.
   While the implementation details vary, tools like [**RabbitMQ**](using-masstransit-with-rabbitmq-and-azure-service-bus),
   [**Amazon SQS and SNS**](complete-guide-to-amazon-sqs-and-amazon-sns-with-masstransit),
   or [**Azure Service Bus**](messaging-made-easy-with-azure-service-bus)
   can significantly improve system resilience and decoupling.

3. Network communication introduces new failure modes, so I always implement [**resilience patterns**](building-resilient-cloud-applications-with-dotnet).
   Here's how I use Polly with resilience pipelines:

```csharp
using Polly;

var pipeline = new ResiliencePipelineBuilder<HttpResponseMessage>()
    .AddRetry(new RetryStrategyOptions
    {
        ShouldHandle = new PredicateBuilder().Handle<ConflictException>(),
        Delay = TimeSpan.FromSeconds(1),
        MaxRetryAttempts = 2,
        BackoffType = DelayBackoffType.Exponential,
        UseJitter = true
    })
    .AddTimeout(new TimeoutStrategyOptions
    {
        Timeout = TimeSpan.FromSeconds(10)
    })
    .Build();

var response = await pipeline.ExecuteAsync(
    async ct => await _httpClient.GetAsync($"http://bookings-service/api/bookings/{bookingId}", ct),
    cancellationToken);
```

Transitioning to HTTP-based communication always brings new problems to solve.
Serialization, error handling, and [**API versioning**](api-versioning-in-aspnetcore) become crucial.
I've learned to plan for these aspects from the start to avoid headaches down the line.

## Implementing an API Gateway

As you extract more services, managing the communication between clients and your microservices can become complex.
An API Gateway can help manage this complexity by providing a single entry point for all clients.

YARP (Yet Another Reverse Proxy) is an excellent tool for [**implementing an API Gateway**](implementing-an-api-gateway-for-microservices-with-yarp) in .NET.
Here's how I typically set it up:

```csharp
var builder = WebApplication.CreateBuilder(args);

builder.Services.AddReverseProxy()
    .LoadFromConfig(builder.Configuration.GetSection("ReverseProxy"));

var app = builder.Build();

app.MapReverseProxy();

app.Run();
```

Then, I configure the routing in `appsettings.json`:

```json
{
  "ReverseProxy": {
    "Routes": {
      "bookings-route": {
        "ClusterId": "bookings-cluster",
        "Match": {
          "Path": "/api/bookings/{**catch-all}"
        }
      },
      "payments-route": {
        "ClusterId": "payments-cluster",
        "Match": {
          "Path": "/api/payments/{**catch-all}"
        }
      }
    },
    "Clusters": {
      "bookings-cluster": {
        "Destinations": {
          "destination1": {
            "Address": "http://bookings-service/"
          }
        }
      },
      "payments-cluster": {
        "Destinations": {
          "destination1": {
            "Address": "http://payments-service/"
          }
        }
      }
    }
  }
}
```

This setup routes requests to the appropriate microservice based on the path.
For example, any request to `/api/bookings/*` will be routed to the `bookings-service`.

<Image
  src="blogs/mnw_109/modular_monolith_extraction_api_gateway.png"
  width={1050}
  height={737}
  alt="Modular monolith extraction process into microservices with an API gateway introduced."
/>

In my projects, I've found that an API Gateway often becomes a critical point for implementing cross-cutting concerns.
As your architecture evolves, consider adding features like:

- [**Rate limiting**](advanced-rate-limiting-use-cases-in-dotnet) to protect your services from overload
- [**Caching**](caching-in-aspnetcore-improving-application-performance) to improve response times for frequently requested data
- Request/response transformation to adapt your internal APIs for external consumption

Start simple, but be prepared to evolve your gateway as you learn more about your system's needs and usage patterns.

## Data Migration Strategy

Data migration is often the trickiest part of moving from a modular monolith to microservices. In my experience, there are two main approaches:

1. **The "One and Done" Approach**

   For simpler systems or those that can handle some downtime, I use this method:

   - Create a new database for the microservice
   - Copy the relevant schema and data from the monolith
   - Switch the application to use the new database

   It's quick and simple but requires a brief downtime.

2. **The Synchronization Approach**

   For complex systems needing minimal disruption, I use this method:

   - Copy the schema and initial data to the new database
   - Set up a sync mechanism (like [change data capture](https://en.wikipedia.org/wiki/Change_data_capture)) to keep the new database updated
   - Gradually shift traffic to the new microservice
   - Eventually, remove the old schema from the monolith

   This approach is more complex but allows for a smoother transition.

In both cases, I always ensure I have a solid backup and rollback plan.
The key is to choose the method that best fits your system's needs and tolerance for complexity.

Remember, if your modular monolith already uses [**separate database schemas for each module**](modular-monolith-data-isolation), you're starting with an advantage.
This logical isolation makes the migration process significantly easier.

## Summary

Migrating from a modular monolith to microservices is a journey I've taken several times, and it's always both challenging and rewarding.
The process we've covered here - from preparing your monolith and choosing the right module to extract
to implementing proper communication patterns - forms the foundation of a successful migration.

Don't rush the process.
Careful planning and incremental changes are key.
Each step teaches you something valuable about your system.

While we've covered the fundamentals, there's always more to learn. I recommend exploring these advanced topics:

- [**Service discovery**](service-discovery-in-microservices-with-net-and-consul) and API gateway patterns
- [**Monitoring and observability**](introduction-to-distributed-tracing-with-opentelemetry-in-dotnet) in a microservices environment
- Advanced [**deployment strategies**](how-to-build-ci-cd-pipeline-with-github-actions-and-dotnet), like blue-green deployment

If you're ready to master this process and gain hands-on experience, I've put together a comprehensive
[**Modular Monolith Architecture**](/modular-monolith-architecture) course.
There's an entire chapter dedicated to extracting modules and moving to microservices.
You'll gain practical skills to confidently navigate your microservices transformation.

Remember, the goal isn't to blindly adopt microservices but to evolve your architecture to best serve your business needs.
Sometimes, a well-structured modular monolith is the right solution.
Other times, a full microservices architecture is the way to go.
The key is understanding the trade-offs and making informed decisions.

Good luck out there, and I'll see you next week.

---



<Sponsorships>

[**Postman v11**](https://blog.postman.com/announcing-postman-v11/) takes your dev collaboration to next level using AI and APIs.
Check out [**Postman v11**](https://blog.postman.com/announcing-postman-v11/) to get started with it's features designed to help bring API collaboration to the next level.

Deliver a great UI with easily customizable [**Page Templates and Building Blocks**](https://www.telerik.com/page-templates-and-ui-blocks?utm_medium=cpm&utm_source=milanjonanovich&utm_campaign=dt_design_templates_building_blocks).
Utilize a collection of over 50 building blocks and 10 page templates designed to help you build modern UI in no time.
All you need to do is copy and paste!
[**Learn more here**](https://www.telerik.com/page-templates-and-ui-blocks?utm_medium=cpm&utm_source=milanjonanovich&utm_campaign=dt_design_templates_building_blocks).

</Sponsorships>

So, I've got this blog I've been running for a couple of years now.
It's a Next.js and TypeScript static site, all optimized for SEO.
Sounds pretty standard, right?
Well, it was, until I ran into a problem.

As my content grew, finding specific articles became a real pain.
It wasn't just me - readers were struggling too.
What's the point of writing all this stuff if no one can find it when they need it?

That's when I decided to look into full-text search.
Now, implementing full-text search on a static site isn't exactly straightforward.
It's not like you can just slap a database on there and call it a day.
You've got to get creative.

In this article, I'll walk you through how I turned my site's search from useless to actually functional.

## The Old Search Approach

Here's how the search function worked before:

1. I would generate a JSON document at build time containing each blog post's metadata.
   This document contains the data needed to perform the search and render the results.
2. When searching the articles, I would fetch the JSON document on the client and perform the search.
   Since this document changes only once a week, I can cache it on the client to improve performance.

Here's the code that would generate the search data JSON document:

```ts
export const generateSearchData = () => {
  const posts = getAllPosts();

  const searchData = posts.map((post) => ({
    slug: post.meta.slug,
    title: post.meta.title,
    excerpt: post.meta.excerpt,
    coverImage: post.meta.coverImage,
    date: post.meta.date,
    searchableContent:
      `${post.meta.title} ${post.meta.excerpt} ${post.content}`.toLowerCase()
  }));

  fs.mkdirSync('./search', { recursive: true });
  fs.writeFileSync(
    path.join(process.cwd(), 'search', 'search-data.json'),
    JSON.stringify(searchData)
  );
};
```

And then the search function is pretty simple (but not very smart):

```typescript
export const search(keyword) = () => {
  const res = await fetch('/search/search-data.json');

  const searchData = await res.json();

  return searchData.filter((p) =>
    p.searchableContent.includes(keyword.toLowerCase())
  );
}
```

This approach has a few issues:

- The search data JSON document is large (861KB) and will continue to grow
- It's slow on large datasets and won't scale well
- It uses an exact match to find results
- It doesn't rank results by relevance

I wasn't happy with this, so here's how I implemented a much better solution.

## Introducing Full-Text Search

To fix these problems, I turned to **full-text search** using [Lunr.js](https://lunrjs.com/).
But what exactly is full-text search?

Full-text search is a technique that allows fast and efficient searching of large volumes of text by creating an index of all words in a document collection
and returning ranked results based on relevance to the search query.

Full-text search works by:

1. **Indexing**: Breaking down text into individual words (tokens)
2. **Stemming**: Reducing words to their base form (e.g., "running" to "run")
3. **Ranking**: Scoring results based on relevance

Another common operation is stop word removal.
Common words that don't add much meaning (like "the", "and", "is") are often removed to save space and improve relevance.

At the heart of full-text search is an [inverted index](https://en.wikipedia.org/wiki/Inverted_index).
It's a data structure that maps each unique term to the documents containing it.
It's "inverted" because it goes from terms to documents, rather than documents to terms.

When searching, results are often ranked using [TF-IDF](https://en.wikipedia.org/wiki/Tf-idf) (Term Frequency-Inverse Document Frequency).
This gives higher scores to terms that are frequent in a document but rare across all documents.

That's the rundown on full-text search. Let's see how to implement it.

## Implementing Full-Text Search

I chose Lunr.js because it's lightweight and works great for static websites. No server required.

I updated the `generateSearchData` function to create and store a full-text search index.
This function runs once at build time, so it's not expensive.

```typescript {6-14,19-22}
export const generateSearchData = () => {
  const posts = getAllPosts();

  // Generate search data as before.

  const index = lunr(function () {
    this.ref('slug');
    this.field('title', { boost: 10 });
    this.field('content', { boost: 5 });

    searchData.forEach((doc) => {
      this.add(doc);
    });
  });

  // Store search data as before.

  // And store the search index.
  fs.writeFileSync(
    path.join(process.cwd(), 'search', 'search-index.json'),
    JSON.stringify(index)
  );
};
```

I'm **boosting** the title with a factor of 10 and the content with a factor of 5.
If the search term matches the title, it will have a higher relevance score.

The downside (for now) is that the index file is big, 2.5MB.
If I were to download this on the client every time, the costs would quickly add up.
I have 100,000 unique visitors per month, which would equate to `250GB` of network egress.

Now, I have to update the search function to use the full-text index:

```typescript {6}
export const search(keyword) = () => {
  const res = await fetch('/search/search-data.json');
  const searchData = await res.json();

  const res = await fetch('/search/search-index.json');
  const searchIndex = lunr.Index.load(indexJson);

  return searchIndex
    .search(keyword)
    .map((result) => {
      const post = searchData?.find((post) => post.slug === result.ref);
      return post ? { ...post, score: result.score } : null;
    })
    .filter((result) => result !== null)
    .sort((a, b) => b.score - a.score);
}
```

What's happening here:

- We're loading the search index from a JSON document using `lunr.Index.load`
- The index has a `search` function allowing us to perform full-text search
- We're expanding the result to also include the relevance score

This allows me to sort the search results based on relevance instead of chronologically.

Here's what the old search implementation returns when searching for "resilience":

<div className="bordered centered">
  <Image
    src="blogs/mnw_108/old_search.png"
    width={814}
    height={912}
    alt="The web page showing search results for the keyword 'resilience' with the most relevant result highlighted."
  />
</div>

The most relevant article is at the fourth spot, sorted chronologically.
You can see how this isn't a good user experience.

But with full-text search, when you search for a keyword like "resilience", you get nicer results:

<div className="bordered centered">
  <Image
    src="blogs/mnw_108/full_text_search.png"
    width={814}
    height={863}
    alt="The web page showing full-text search results for the keyword 'resilience' with the most relevant result highlighted."
  />
</div>

I also added a relevance score next to each article because it looks cool.

## Optimizing The Search

Now, I have a powerful search, but the index was huge (a whopping 2.5MB) and will continue to grow.
The search data file is also 861KB.

Enter [Brotli](https://en.wikipedia.org/wiki/Brotli) compression:

- I compressed the search index and data using Brotli on the server side
- In the browser, I decompress the files before using them to perform the search

```typescript {8,12}
import { compress } from 'brotli';

export const generateSearchData = () => {
  // Create the search data and index

  fs.writeFileSync(
    path.join(process.cwd(), 'search', 'search-index.br'),
    compress(Buffer.from(JSON.stringify(index)))
  );
  fs.writeFileSync(
    path.join(process.cwd(), 'search', 'search-data.br'),
    compress(Buffer.from(JSON.stringify(searchData)))
  );
};
```

The results:

- 2.5MB → 193KB **(-92%)**
- 861KB → 180KB **(-79%)**

If you want to learn more about compression algorithms, check out this [**article about response compression**](response-compression-in-aspnetcore).

After fetching the compressed search data and full-text index on the client, I cache them for subsequent searches.

The searches are lightning-fast now, and the results are more relevant.

## Server-Side Options for Full-Text Search

While Lunr.js works great for my static site, larger apps with more data need a robust full-text search solution.
So, here are some some server-side options you could explore.

[Lucene](https://lucene.apache.org/) is the foundation of many search engines.
It's written in Java but has ports to other languages.
Lucene provides robust full-text indexing and search capabilities.
It is also highly efficient and customizable, making it a popular choice for developers who need fine-grained control over their search functionality.

[Apache Solr](https://solr.apache.org/) builds on top of Lucene, offering additional features like distributed indexing, replication, and load-balanced querying.
Solr includes powerful capabilities such as faceting and highlighting, which can greatly enhance the search experience.

If you're already using [PostgreSQL](https://www.postgresql.org/), the built-in [full-text search](https://www.postgresql.org/docs/current/textsearch.html) is worth considering.
PostgreSQL uses its own text search engine, which supports multiple languages and custom dictionaries.
While not as feature-rich as dedicated search engines, it can be a convenient option for applications that want to keep their stack simple.

I've been tinkering with PostgreSQL [full-text search using EF Core](https://www.npgsql.org/efcore/mapping/full-text-search.html).
Here's what a full-text search query looks like:

```csharp
var blogs = context
    .BlogPosts
    .Where(b =>
        EF.Functions.ToTsVector("english", b.Title + " " + b.Excerpt + " " + b.Content)
            .Matches(EF.Functions.PhraseToTsQuery("english", searchTerm)))
    .Select(b => new
    {
        b.Slug,
        b.Title,
        b.Excerpt,
        b.Date,
        Rank = EF.Functions.ToTsVector("english", b.Title + " " + b.Excerpt + " " + b.Content)
            .Rank(EF.Functions.PhraseToTsQuery("english", searchTerm))
    })
    .OrderByDescending(b => b.Rank)
    .ToList();
```

But I'll have to write a dedicated article to cover this, so let's wrap it up.

## Wrapping Up

The old search functionality was, frankly, embarrassing.

It was slow, dumb as a rock, and about as useful as a chocolate teapot.

Now? It's actually not half bad.

- Searches are lightning-fast
- Results are more relevant
- The user experience is much smoother

I'm not gonna lie, there was a moment of frustration when I thought about giving up.
But I'm glad I stuck with it.
There's something satisfying about building it yourself, you know?

It might not be pretty, but it works.
And sometimes, that's all that matters.

The Brotli compression bit was a pleasant surprise.
I thought I'd end up with a massive index file that would make mobile users cry.
But nope, it all works smoothly.
Go figure.

Have you tried implementing search on your site? How'd it go?

That's all for today.

See you next week.

---



<Sponsorships>

Introducing [**IronSecureDoc**](https://ironsoftware.com/enterprise/securedoc/?utm_source=milan&utm_medium=email&utm_campaign=milan) -
the PDF security & compliance server.
Protect your PDFs with IronSecureDoc, the latest from Iron Software.
Redact sensitive data, add digital signatures, and stay compliant with PDF/A & PDF/UA standards.
Easy integration with our REST API.
[**Learn more here**](https://ironsoftware.com/enterprise/securedoc/?utm_source=milan&utm_medium=email&utm_campaign=milan).

Take Postman wherever you work with the [**Postman GA VS Code extension**](https://learning.postman.com/docs/getting-started/basics/about-vs-code-extension)!
The Postman VS Code extension enables you to develop and test your APIs in Postman directly from Visual Studio Code and Visual Studio Code Insiders.
[**Try it out**](https://learning.postman.com/docs/getting-started/basics/about-vs-code-extension)!

</Sponsorships>

I've been tinkering with .NET Aspire lately, and I've got some thoughts to share.
If you're curious about this new cloud-native development tool from Microsoft, stick around.
I'll break down what's great, what's not, and how you can start using it.

> .NET Aspire is an opinionated, cloud-ready stack for building observable, production-ready, distributed applications.

[**.NET Aspire**](https://learn.microsoft.com/en-us/dotnet/aspire/get-started/aspire-overview) is Microsoft's latest offering for cloud-native application development.
It aims to simplify the process of building, deploying, and managing distributed applications.

Distributed applications often consist of small applications that consume external services like databases, message brokers, and caching.
.NET Aspire gives you a set of tools to make building distributed applications easier.

## .NET Aspire Orchestration

How are you setting up a local development environment?
I often use **Docker Compose** to configure my applications and run external services.
It's a simple setup, but you need to manage environment variables and connection strings.
If you're not familiar with Docker, it can prove to be quite tricky sometimes.

Here's a `docker-compose.yml` file from a recent project:

```yml
services:
  contentplatform-api:
    image: ${DOCKER_REGISTRY-}contentplatform-api
    container_name: ContentPlatform.Api
    build:
      context: .
      dockerfile: ContentPlatform.Api/Dockerfile
    ports:
      - 5000:8080
      - 5001:8081

  contentplatform-reporting-api:
    image: ${DOCKER_REGISTRY-}contentplatform-reporting-api
    container_name: ContentPlatform.Reporting.Api
    build:
      context: .
      dockerfile: ContentPlatform.Reporting.Api/Dockerfile
    ports:
      - 6000:8080
      - 6001:8081

  contentplatform-presentation:
    image: contentplatform-ui:latest
    container_name: ContentPlatform.Presentation
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
    ports:
      - 3000:80

  contentplatform-db:
    image: postgres:latest
    container_name: ContentPlatform.Db
    environment:
      - POSTGRES_DB=contentplatform
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
    volumes:
      - ./.containers/db:/var/lib/postgresql/data
    ports:
      - 5432:5432

  contentplatform-mq:
    image: rabbitmq:management
    container_name: ContentPlatform.RabbitMq
    hostname: contentplatform-mq
    volumes:
      - ./.containers/queue/data/:/var/lib/rabbitmq
      - ./.containers/queue/log/:/var/log/rabbitmq
    environment:
      RABBITMQ_DEFAULT_USER: guest
      RABBITMQ_DEFAULT_PASS: guest
```

This sets up two APIs, a client application, PostgreSQL, and RabbitMQ.
I also have to configure the connection strings manually to connect to these services.

So, I decided to migrate this application to .NET Aspire and documented the process.

You can right-click an existing project in Visual Studio and select `Add > .NET Aspire Orchestrator Support...`.

<figure className="figure-center">
  <Image
    src="blogs/mnw_107/aspire_orchestration.png"
    width={488 / 1.4}
    height={706 / 1.4}
    alt="Context menu with '.NET Aspire Orchestrator Support...' highlighted."
  />
  <figcaption>
    Source:
    [Microsoft](https://learn.microsoft.com/en-us/dotnet/aspire/get-started/add-aspire-existing-app)
  </figcaption>
</figure>

This will add an `AppHost` and `ServiceDefaults` project to your solution.
You will then repeat this for the remaining projects in your solution to enlist them all in Aspire orchestration.

The `AppHost` project is responsible for orchestration.
You can define your entire application stack in a single, readable file.
Running the `AppHost` project from Visual Studio will start the required applications and services.

Here's the setup for my application using Aspire:

```csharp
IDistributedApplicationBuilder builder = DistributedApplication.CreateBuilder(args);

var postgres = builder.AddPostgres("contentplatform-db")
    .WithPgAdmin();

var rabbitMq = builder.AddRabbitMQ("contentplatform-mq")
    .WithManagementPlugin();

builder.AddProject<Projects.ContentPlatform_Api>("contentplatform-api")
    .WithReference(postgres)
    .WithReference(rabbitMq);

builder.AddProject<Projects.ContentPlatform_Reporting_Api>("contentplatform-reporting-api")
    .WithReference(postgres)
    .WithReference(rabbitMq);

builder.AddProject<Projects.ContentPlatform_Presentation>("contentplatform-presentation");

builder.Build().Run();
```

The Aspire version is much more concise and readable.
Adding new services or changing configurations is straightforward.
You also get built-in observability.
Aspire includes tools for logging, metrics, and distributed tracing out of the box, making it easier to monitor and debug your applications.

When you run the application, you can see your applications and services on the Aspire dashboard:

<Image
  src="blogs/mnw_107/content_platfrom_resources.png"
  width={1919}
  height={583}
  alt="Aspire dashboard resource views showing the application services and containers running."
/>

### Orchestration - The Bad Parts

There are a few things I don't like with the current Aspire setup.

The `AppHost` project needs to reference all other projects to enlist them in orchestration.
If your services are all in one solution, this might be fine.
But what about large microservices systems?

We can go around this limitation by building a Docker image for an external service.
There's an `AddContainer` method that allows us to configure container resources.
However, we won't be able to debug these services.

The `ServiceDefaults` projects needs to be visible to all other applications.
Again, this works perfectly fine if everything is in one solution.
We can also distribute this project as a NuGet package for complex systems.

## .NET Aspire Integrations

If you're wondering how I configured PostgreSQL and RabbitMQ in the previous example, this is made available using Aspire Integrations.
These are NuGet packages that allow you to integrate with popular services, such as Redis or PostgreSQL.
Aspire integrations take care of many cloud-native concerns for you, like adding health checks and telemetry.

You can right-click on the `AppHost` project and select `Add > .NET Aspire package...` to see the list of available integrations:

<Image
  src="blogs/mnw_107/aspire_integrations.png"
  width={1088}
  height={917}
  alt="NuGet browser view showing a list of .NET Aspire integrations."
/>

If we want to add Redis to our project, we can install the `Aspire.Hosting.Redis` package.
Then, we would configure the Redis integration in the `AppHost` project:

```csharp {5,10}
var builder = DistributedApplication.CreateBuilder(args);

// Other service omitted for brevity

var redis = builder.AddRedis("contentplatform-cache");

builder.AddProject<Projects.ContentPlatform_Api>("contentplatform-api")
    .WithReference(postgres)
    .WithReference(rabbitMq)
    .WithReference(redis);

builder.Build().Run();
```

You can find a list of supported [Aspire integrations](https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/integrations-overview) in the documentation.

When you configure a resource with a known connection string format, Aspire automatically injects an environment variable.
The connection string name will have the same name as the respective resource.

- `WithReference(postgres)` produces `ConnectionStrings__contentplatform-db="<VALUE>"`
- `WithReference(rabbitMq)` produces `ConnectionStrings__contentplatform-mq="<VALUE>"`
- `WithReference(redis)` produces `ConnectionStrings__contentplatform-cache="<VALUE>"`

This lets you use logical connection string names when configuring your services:

```csharp
builder.Services.AddDbContext<ApplicationDbContext>(o =>
    o.UseNpgsql(builder.Configuration.GetConnectionString("contentplatform-db")));
```

## Service Defaults and OpenTelemetry

One of Aspire's killer features is its built-in observability stack.
It integrates [**OpenTelemetry**](introduction-to-distributed-tracing-with-opentelemetry-in-dotnet), providing distributed tracing, metrics, and logging out of the box.

When you enlist a project in .NET Aspire orchestration, there are some updates made to the `Program` file automatically:

- `AddServiceDefaults` is called to configure OpenTelemetry, health checks, and service discovery
- `MapDefaultEndpoints` is called to expose the health check endpoint

```csharp {3,9}
var builder = WebApplication.CreateBuilder(args);

builder.AddServiceDefaults();

// Other code omitted for brevity

var app = builder.Build();

app.MapDefaultEndpoints();

// Other code omitted for brevity

app.Run();
```

You can customize `AddServiceDefaults` according to your requirements.
For example, if you're using MassTransit, you can add the respective tracing configuration for this library.

Here's the distributed traces view on the Aspire dashboard.
You can see a `POST` request hitting the `contentplatform-api` service, publishing an `ArticleCreatedEvent`, and consuming that message in the `contentplatform-reporting-api` service.

<Image
  src="blogs/mnw_107/content_platfrom_traces.png"
  width={1920}
  height={550}
  alt="Aspire dashboard traces views showing one distributed trace."
/>

For local development, the .NET Aspire dashboard provides a UI for viewing telemetry data.
In a production environment, you can configure the OpenTelemetry server to receive telemetry data using the `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable.

## Deploying .NET Aspire Applications

.NET Aspire simplifies the deployment process for distributed applications, especially when targeting Azure.
To deploy an Aspire application, you first generate a **manifest file** using the `dotnet run` command with specific parameters.
This manifest is a JSON file that describes all the resources defined in your Aspire project, including services, databases, and other dependencies.

<figure>
  <Image
    src="blogs/mnw_107/aspire_manifest.png"
    width={3659}
    height={1821}
    alt=".NET Aspire manifest JSON file example."
  />
  <figcaption>
    Source:
    [Microsoft](https://learn.microsoft.com/en-us/dotnet/aspire/deployment/manifest-format)
  </figcaption>
</figure>

Deployment tools can use the manifest to set up the necessary infrastructure in your target environment.
Aspire generates the required configuration for Azure Container Apps or Kubernetes for Azure deployments.
It handles tasks like setting up networking, scaling services, and configuring monitoring automatically.

Here's a simple example of generating a manifest:

```
dotnet run --project ContentPlatform.AppHost\ContentPlatform.AppHost.csproj `
    -- --publisher manifest --output-path ../aspire-manifest.json
```

This command creates a JSON manifest file that deployment tools can use to set up your application in the cloud or on-premises infrastructure.

You can learn more about [Aspire deployment](https://learn.microsoft.com/en-us/dotnet/aspire/deployment/overview) in the documentation.

## Summary

I've used .NET Aspire a lot lately, and I'm genuinely impressed.

Aspire makes building complex systems much easier.
I can set up a distributed system with just a few lines of C# code.
This is much simpler than using Docker Compose.
The built-in observability and monitoring tools are also great.

While .NET Aspire is now production-ready, the ecosystem around it is still growing.
Developers, particularly those new to cloud-native concepts, might face a learning curve.

Should you adopt Aspire in your .NET projects?

If you're building distributed applications, especially for Azure, I'd say give it a try.
However, you might want to evaluate carefully if you work on simpler applications or use non-Azure cloud services.

That's all for today.

See you next week.

**P.S.** You can find the source code for this example in [**this repository**](https://github.com/m-jovanovic/aspire-orchestration).

---



<Sponsorships>

Take Part in [**The State of Designer-Developer Collaboration 2024 Survey**](https://progress.co1.qualtrics.com/jfe/form/SV_1H2T9k9UuXgrkMK?utm_medium=cpm&utm_source=milanjovanovich&utm_campaign=dsk-awareness-design-dev-collab-survey-2024)!
Are you a web developer, a designer, or someone with a stake in the designer-developer collaboration process?
Then take this newly launched global survey.
[**We need your input**](https://progress.co1.qualtrics.com/jfe/form/SV_1H2T9k9UuXgrkMK?utm_medium=cpm&utm_source=milanjovanovich&utm_campaign=dsk-awareness-design-dev-collab-survey-2024)!

Comment on collections, folders, and requests using [**Postman**](https://learning.postman.com/docs/collaborating-in-postman/working-with-your-team/discussing-your-work/)!
Launch into hyper-efficiency using comments to discuss your work with your teammates in Postman.
[**Learn more here**](https://learning.postman.com/docs/collaborating-in-postman/working-with-your-team/discussing-your-work/)!

</Sponsorships>

As a .NET developer, I've spent countless hours working with external APIs.
It's a crucial part of modern software development, but let's be honest - it can be a real pain sometimes.

We've all been there, wrestling with `HttpClient`, writing repetitive code, and hoping we didn't miss a parameter or header somewhere.

That's why I want to introduce you to **Refit**, a library that's been a game-changer for me.

Imagine turning your API into a live interface - sounds too good to be true, right?
But that's exactly what Refit does.
It handles all the HTTP heavy lifting, letting you focus on what matters: your application logic.

In this article, I'll explain how Refit can transform the way you work with APIs in your .NET projects.

## What is Refit?

[Refit](https://github.com/reactiveui/refit) is a type-safe REST library for .NET.
It allows you to define your API as an interface, which Refit then implements for you.
This approach reduces boilerplate code and makes your API calls more readable and maintainable.

You describe your API endpoints using method signatures and attributes, and Refit takes care of the rest.

Let me break down why I find Refit so powerful:

- **Automatic serialization and deserialization**:
  You won't have to convert your objects to JSON and back.
  Refit handles all of that for you.
- **Strongly-typed API definitions**:
  Refit helps you catch errors early.
  If you mistype a parameter or use the wrong data type, you'll know at compile time, not when your app crashes in production.
- **Support for various HTTP methods**:
  GET, POST, PUT, PATCH, DELETE - Refit has you covered.
- **Request/response manipulations**:
  You can add custom headers or handle specific content types in a straightforward way.

But what I appreciate most about Refit is how it promotes clean, readable code.
Your API calls become self-documenting.
Anyone reading your code can quickly understand what each method does without diving into implementation details.

## Setting Up and Using Refit in Your Project

Let's set up Refit and see it in action using the [JSONPlaceholder API](https://jsonplaceholder.typicode.com/).
We'll implement a full CRUD interface and demonstrate its usage in a [**Minimal API application**](automatically-register-minimal-apis-in-aspnetcore).

First, install the required NuGet packages:

```powershell
Install-Package Refit
Install-Package Refit.HttpClientFactory
```

Now, let's create our Refit interface:

```csharp
using Refit;

public interface IBlogApi
{
    [Get("/posts/{id}")]
    Task<Post> GetPostAsync(int id);

    [Get("/posts")]
    Task<List<Post>> GetPostsAsync();

    [Post("/posts")]
    Task<Post> CreatePostAsync([Body] Post post);

    [Put("/posts/{id}")]
    Task<Post> UpdatePostAsync(int id, [Body] Post post);

    [Delete("/posts/{id}")]
    Task DeletePostAsync(int id);
}

public class Post
{
    public int Id { get; set; }
    public string Title { get; set; }
    public string Body { get; set; }
    public int UserId { get; set; }
}
```

We define our `IBlogApi` interface with methods for all CRUD operations: GET (single and list), POST, PUT, and DELETE.
The `Post` class represents the structure of our blog posts.

Then you have to register Refit in your dependency injection container:

```csharp {4}
using Refit;

builder.Services
    .AddRefitClient<IBlogApi>()
    .ConfigureHttpClient(c => c.BaseAddress = new Uri("https://jsonplaceholder.typicode.com"));
```

Finally, we can use `IBlogApi` in our Minimal API endpoints:

```csharp
app.MapGet("/posts/{id}", async (int id, IBlogApi api) =>
    await api.GetPostAsync(id));

app.MapGet("/posts", async (IBlogApi api) =>
    await api.GetPostsAsync());

app.MapPost("/posts", async ([FromBody] Post post, IBlogApi api) =>
    await api.CreatePostAsync(post));

app.MapPut("/posts/{id}", async (int id, [FromBody] Post post, IBlogApi api) =>
    await api.UpdatePostAsync(id, post));

app.MapDelete("/posts/{id}", async (int id, IBlogApi api) =>
    await api.DeletePostAsync(id));
```

What I love about this setup is its simplicity.
We've created a fully functional API that communicates with an external service, all in just a few lines of code.
No manual HTTP requests, no raw JSON handling - Refit takes care of all that for us.

## Query Parameters and Route Binding

When working with APIs, you often need to send data as part of the URL, either in the route or as query parameters.
Refit makes this process simple and type-safe.

Let's extend our `IBlogApi` interface with some more complex scenarios:

```csharp {6,8}
public interface IBlogApi
{
    // Other methods omitted for brevity

    [Get("/posts")]
    Task<List<Post>> GetPostsAsync([Query] PostQueryParameters parameters);

    [Get("/users/{userId}/posts")]
    Task<List<Post>> GetUserPostsAsync(int userId);
}

public class PostQueryParameters
{
    public int? UserId { get; set; }
    public string? Title { get; set; }
}
```

Let's break this down:

- `GetPostsAsync` uses an object to represent query parameters.
  This approach is excellent for endpoints with many optional parameters.
  Refit will automatically convert this object into a query string.
- `GetUserPostsAsync` demonstrates passing in route parameters (`userId`) directly.

Using an object for query parameters makes your code type-safe and [**refactoring-friendly**](5-awesome-csharp-refactoring-tips).
If you need to add a new query parameter, you just add a property to `PostQueryParameters`.
Your existing code won't break, and your IDE can help you discover the new options.

## Dynamic Headers and Authentication

Another common requirement when integrating with APIs is including custom headers or authentication tokens with your requests.
Refit provides several ways to handle this, from simple static headers to dynamic, request-specific authentication.

Let's explore some scenarios:

```csharp
public interface IBlogApi
{
    [Headers("User-Agent: MyAwesomeApp/1.0")]
    [Get("/posts")]
    Task<List<Post>> GetPostsAsync();

    [Get("/secure-posts")]
    Task<List<Post>> GetSecurePostsAsync([Header("Authorization")] string bearerToken);

    [Get("/user-posts")]
    Task<List<Post>> GetUserPostsAsync([Authorize(scheme: "Bearer")] string token);
}
```

- You can add a static header to all requests by using the `Headers` attribute
- With the `Header` attribute, you can pass a header value dynamically as a parameter
- The `Authorize` attribute is a convenient way to add Bearer token authentication

But what if you need to add the same dynamic header to all requests?

That's where `DelegatingHandler` comes in handy.

You can learn more about [**using delegating handlers in this article**](extending-httpclient-with-delegating-handlers-in-aspnetcore).

I've found delegating handlers especially helpful in providing API keys, which are typically static.

## JSON Serialization Options

Refit gives you flexibility when choosing and configuring your JSON serializer.
By default, Refit uses `System.Text.Json`, is the built-in JSON serializer in modern .NET versions.

However, you can easily switch to `Newtonsoft.Json` if you need its features.

Here's how you can configure Refit to use it.

First, install the Newtonsoft.Json support package:

```powershell
Install-Package Refit.Newtonsoft.Json
```

Then, configure Refit to use Newtonsoft.Json:

```csharp {7-11}
using Newtonsoft.Json;
using Newtonsoft.Json.Serialization;
using Refit;

builder.Services.AddRefitClient<IBlogApi>(new RefitSettings
{
    ContentSerializer = new NewtonsoftJsonContentSerializer(new JsonSerializerSettings
    {
        ContractResolver = new CamelCasePropertyNamesContractResolver(),
        NullValueHandling = NullValueHandling.Ignore
    })
})
.ConfigureHttpClient(c => c.BaseAddress = new Uri("https://jsonplaceholder.typicode.com"));
```

This setup uses camel case for property names and ignores null values when serializing.

`System.Text.Json` is faster and uses less memory, making it a great default choice.
However, `Newtonsoft.Json` offers more features and might be necessary for compatibility with older systems or specific serialization needs.

## Handling HTTP Responses

While Refit's default behavior of automatically deserializing responses into your defined types is convenient, there are times when you need more control over the HTTP response.

Refit provides two options for these scenarios:
[`HttpResponseMessage`](https://learn.microsoft.com/en-us/dotnet/api/system.net.http.httpresponsemessage) and `ApiResponse<T>`.

Let's update the `IBlogApi` to use these types:

```csharp
public interface IBlogApi
{
    [Get("/posts/{id}")]
    Task<HttpResponseMessage> GetPostRawAsync(int id);

    [Get("/posts/{id}")]
    Task<ApiResponse<Post>> GetPostWithMetadataAsync(int id);

    [Post("/posts")]
    Task<ApiResponse<Post>> CreatePostAsync([Body] Post post);
}
```

Now, let's break down how to use these, starting with `HttpResponseMessage`:

```csharp {1}
HttpResponseMessage response = await blogApi.GetPostRawAsync(1);

if (response.IsSuccessStatusCode)
{
    var content = await response.Content.ReadAsStringAsync();
    var post = JsonSerializer.Deserialize<Post>(content);
    Console.WriteLine($"Retrieved post: {post.Title}");
}
else
{
    Console.WriteLine($"Error: {response.StatusCode}");
}
```

This approach gives you full control over the HTTP response. You can access status codes, headers, and the raw content.
But you will have to deal with deserialization manually.

`ApiResponse<T>` is a Refit-specific type that wraps the deserialized content and response metadata.
It's a great middle ground when you need the typed response and access to headers or status codes.

Here's a more complex example using `ApiResponse<T>` for creating a post:

```csharp {3}
var newPost = new Post { Title = "New Post", Body = "Content", UserId = 1 };

ApiResponse<Post> createResponse = await blogApi.CreatePostAsync(newPost);

if (createResponse.IsSuccessStatusCode)
{
    var createdPost = createResponse.Content;
    var locationHeader = createResponse.Headers.Location;
    Console.WriteLine($"Created post with ID: {createdPost.Id}");
    Console.WriteLine($"Location: {locationHeader}");
}
else
{
    Console.WriteLine($"Error: {createResponse.Error.Content}");
    Console.WriteLine($"Status: {createResponse.StatusCode}");
}
```

This approach allows you to access the created resource, check specific headers like `Location`, and handle errors gracefully.

## Takeaway

Refit transforms the way we interact with APIs in .NET applications.
Converting your API into a strongly typed interface simplifies your code, enhances type safety, and improves maintainability.

The key Refit benefits we've explored include:

- Simplified API calls with automatic serialization and deserialization
- Flexible parameter handling for complex queries
- Easy management of headers and authentication
- Options for JSON serialization to fit your project's needs
- Granular control over HTTP responses when required

In my experience, Refit shines in projects of all sizes.
I've used it in small applications and large-scale microservices architectures.
It eliminates boilerplate code, reduces the risk of errors, and allows you to focus on your application's core logic rather than the intricacies of HTTP communication.

Remember, while Refit makes API interactions more straightforward, it's not a substitute for understanding the underlying principles of RESTful communication and HTTP.

That's all for today. Stay awesome, and I'll see you next week.

**P.S.** You can find the source code for this example in [**this repository**](https://github.com/m-jovanovic/refit-client-example).

---



<Sponsorships>

[**DDAL/EF**](https://ddal.io/), or **Dynamic Data Analysis Language** for Entity Framework Core,
is a powerful tool and simple language to retrieve formatted JSON data from a relational database.
You can use it to filter your data, retrieve complex lists and grids, and to fetch formatted data for your charts.
No SQL, no joins, no fuss: a flat syntax for simple yet powerful operations.
[**Learn more**](https://ddal.io/).

Reuse commonly used scripts and tests to packages in your team's [**Package Library**](https://learning.postman.com/docs/tests-and-scripts/write-scripts/package-library/)
using Postman!
Add commonly-used scripts and tests to packages in your team's Package Library, and reuse them in your personal, private, and team workspaces using Postman!
[**Learn more here**](https://learning.postman.com/docs/tests-and-scripts/write-scripts/package-library/).

</Sponsorships>

I've been coding in .NET for years, but I never built an event sourced system.
Event sourcing has always intrigued me, though.
The idea of capturing every change and having a complete history of your data - it's fascinating.

So, I decided to dive in.
Not as an expert but as a curious developer.

In this newsletter, I'm sharing my journey into event sourcing.

- What is it really?
- Why does it matter?
- And how might it change the way we think about our .NET apps?

We'll look at the core concepts of event sourcing, potential benefits, and even some practical examples.

## What is Event Sourcing?

> Event Sourcing is an architectural design pattern where changes that occur in a domain are immutably stored as events in an append-only log.

_&mdash; [Event Store](https://www.eventstore.com/event-sourcing)_

When I first encountered event sourcing, it seemed complex.
But stripped down, it's a surprisingly simple idea: store changes, not just the current state.

Think of a bank account or wallet.
Normally, we'd just save the balance.
With event sourcing, we record every deposit and withdrawal.
The balance is then calculated from these events.

This diagram illustrates the difference:

<Image
  src="blogs/mnw_105/event_sourcing.png"
  width={2695}
  height={1467}
  alt="Event sourcing comparison to traditional data storage."
/>

This shift from storing state to storing events is the essence of event sourcing.
It's like keeping a detailed diary of your application's data rather than just a snapshot.
It's not just about where you are but how you got there.
For me, this was a lightbulb moment.

## Why Use Event Sourcing?

As I dug deeper into event sourcing, I kept asking myself: "Why would I use this instead of traditional data storage?"

Here's what I've discovered:

- **Full Audit Trail**: Every change is recorded.
  This is huge for businesses dealing with sensitive data or financial transactions.
  Imagine being able to trace every step of an order's journey or every modification to a user's account.
- **Debugging Time Machine**: With event sourcing, you can reconstruct the state of your application at any point in time.
  As a developer, this feels like a superpower.
  Tracking down bugs becomes less about guesswork and more about replay.
- **Business Insights**: All those stored events?
  They're a goldmine of data.
  You can analyze patterns, user behavior, or system performance in ways that might be impossible with just current-state data.
- **Flexibility**: Need to add a new feature that requires historical data?
  With event sourcing, it's already there.
  This flexibility could have saved me from many "I wish we had kept that information" moments.

Real-world use cases for event sourcing started to make sense:

- E-commerce platforms leveraging it for order tracking and inventory management.
- Financial systems use it for accurate transaction histories.
- IoT applications use it to analyze sensor data over time.

While it's not a silver bullet (what is in programming?), I'm beginning to see why so many developers are excited about event sourcing.
It's not just about storing data; it's about intent and behavior.

## Core Concepts And Practical Examples

As I started to implement event sourcing, understanding the core concepts became much easier with a concrete example.
Let's walk through a simple bank account scenario to see how event sourcing works.

### Events

Events are immutable records of something that happened.
In our bank account example, we might have events like `AccountOpened`, `MoneyDeposited`, and `MoneyWithdrawn`.

Here's how we might define these in C#:

```csharp
public record AccountOpened(Guid AccountId, DateTime OpenedAt);
public record MoneyDeposited(decimal Amount, DateTime DepositedAt);
public record MoneyWithdrawn(decimal Amount, DateTime WithdrawnAt);
```

[**Records**](records-anonymous-types-non-destructive-mutation) are a perfect fit for events, as they are immutable by design.

### State

In event sourcing, the current state is calculated by applying all events in order.

Here's how our `Account` class looks:

```csharp
public class Account
{
    public Guid Id { get; private set; }
    public decimal Balance { get; private set; }

    private List<object> _events = new List<object>();

    public Account(Guid id)
    {
        ApplyEvent(new AccountCreated(id));
    }

    public void Deposit(decimal amount)
    {
        ApplyEvent(new MoneyDeposited(amount));
    }

    public void Withdraw(decimal amount)
    {
        if (Balance >= amount)
        {
            ApplyEvent(new MoneyWithdrawn(amount));
        }
        else
        {
            throw new InvalidOperationException("Insufficient funds");
        }
    }

    private void ApplyEvent(object @event)
    {
        _events.Add(@event);

        switch (@event)
        {
            case AccountCreated e:
                Id = e.AccountId;
                Balance = 0;
                break;
            case MoneyDeposited e:
                Balance += e.Amount;
                break;
            case MoneyWithdrawn e:
                Balance -= e.Amount;
                break;
        }
    }
}
```

Notice how the `Account` class maintains its state.
Each method (`Deposit`, `Withdraw`) doesn't directly modify the balance.
Instead, it creates and applies an event.
The `ApplyEvent` method then updates the state based on these events.

### Event Store

In our simple example, we're using a list (`_events`) to store events.
In a real system, we would persist these events in a database.
The key principle remains: events are appended, never modified.

For production systems, there are specialized event sourcing databases like [EventStoreDB](https://www.eventstore.com/).

There's also [**Marten**](fast-document-database-in-net-with-marten), a .NET library that adds document database and event sourcing capabilities to PostgreSQL.

## Putting It All Together

Here's how we might use our event sourced `Account`:

- An action (like depositing money) triggers the creation of an event.
- The event is stored in the event store (in our simple example, it's just added to the `_events` list).
- The event is applied to update the current state of the `Account`.
- We can rebuild the state by replaying all events in order when needed.

```csharp
var account = new Account(Guid.NewGuid());
account.Deposit(100);
account.Withdraw(30);
account.Deposit(50);

Console.WriteLine($"Final balance: {account.Balance}"); // Output: Final balance: 120
```

We'd store these events in a database in a real event sourcing system.
This allows us to replay the events on demand to produce the current state.

## Challenges and Considerations

Since I started researching event sourcing, I've seen its potential and its hurdles.

Event sourcing itself is a simple idea.
However, the underlying complexity of this approach concerns me.
There's a significant learning curve from event sourcing basics to _applying event sourcing in production_.

It's not just about storing data differently.
It's a fundamental shift in how you model and think about your domain.
This complexity extends to the infrastructure level.

Performance is another consideration that's often overlooked.
While appending events is typically fast, reconstructing the current state from a long history of events can be slow.
Real-world systems often need to implement caching strategies or snapshots to mitigate this.
Event sourcing is also eventually consistent on the read side.

One of the trickiest aspects I've encountered is event schema evolution (event versioning).
As your system grows and changes, so will your events.
Managing these changes without breaking existing event streams is a challenge that requires careful planning and design.
I'm still researching best practices.

## In Summary

Event sourcing has a steep learning curve, even for an experienced developer.
It requires a fundamental shift in how you think about data and system design.

If you want to give it a try, start small.
Implement a simple event-sourced system in a side project.
It's the best way to grapple with the concepts hands-on.
As you do, you might find that Domain-Driven Design (DDD) principles align well with event sourcing.

Remember, the goal isn't to use event sourcing everywhere but to understand where it can add value.

If you're ready to explore this topic further, check out [**Modular Monolith Architecture**](/modular-monolith-architecture).
There's an entire chapter on Event-Driven Architecture, which directly complements what you've learned about event sourcing here.

In a future newsletter, we'll explore a more real-world application of event sourcing.

Good luck out there, and I'll see you next week.

---



<Sponsorships>

Stop coding common elements!
[**Syncfusion**](https://www.syncfusion.com/?utm_source=milanjovanovicnewsletter&utm_medium=text&utm_campaign=esee_milanjovanovic_newsletter)
offers a developer's dream-over 1,800 prebuilt UI components for web, mobile, and desktop platforms.
Focus on your app's unique features, not repetitive UI tasks.
A free community license is also available for individual developers and startups.
[**Try Syncfusion for free today**](https://www.syncfusion.com/?utm_source=milanjovanovicnewsletter&utm_medium=text&utm_campaign=esee_milanjovanovic_newsletter).

Collaborate directly with your external partners in Postman!
You can use [**Partner Workspaces**](https://learning.postman.com/docs/collaborating-in-postman/using-workspaces/partner-workspaces/)
to help your partners consume your team's APIs and work together on API projects.
[**Check out Postman's Partner Workspaces**](https://learning.postman.com/docs/collaborating-in-postman/using-workspaces/partner-workspaces/),
now available for Professional and Enterprise plans in Postman v11.

</Sponsorships>

If you were to glance at the folder structure of your system, could you tell what the system is about?
And here's a more interesting question.
Could a new developer on your team easily understand what the system does based on the folder structure?

Your architecture should communicate what problems it solves.
Organizing your system around use cases leads to a structure aligned with the business domain.
This approach is called **screaming architecture**.

[Screaming architecture](https://blog.cleancoder.com/uncle-bob/2011/09/30/Screaming-Architecture.html) is a term coined by Robert Martin (Uncle Bob).
He argues that a software system's structure should communicate what the system is about.
He draws a parallel between looking at a blueprint for a building, where you can tell the purpose of the building based on the blueprint.

In this article, I want to show some practical examples and discuss the benefits of screaming architecture.

## A Use Case Driven Approach

A use case represents a specific interaction or task that a user wants to achieve within your system.
It encapsulates the business logic required to fulfill that task.
A use case is a high-level description of a user's goal.
For example, "reserving an apartment" or "purchasing a ticket".
It focuses on the _what_ of the system's behavior, not the _how_.

When you look at the folder structure and source code files of your system:

- Do they scream: Apartment Booking System or Ticketing System?
- Or do they scream ASP.NET Core?

Here's an example of a folder structure organized around technical concerns:

```powershell
📁 Api/
|__ 📁 Controllers
|__ 📁 Entities
|__ 📁 Exceptions
|__ 📁 Repositories
|__ 📁 Services
    |__ #️⃣ ApartmentService.cs
    |__ #️⃣ BookingService.cs
    |__ ...
|__ 📁 Models
```

Somewhere inside these folders, we'll find concrete classes that contain the system's behavior.
You'll notice that the cohesion with this folder structure is low.

How does screaming architecture help?

A use case driven approach will place the system's use cases as the top-level concept.
I also like to group related use cases into a top-level feature folder.
Inside a use case folder, we may find technical concepts required to implement it.

[**Vertical slice architecture**](vertical-slice-architecture) also approaches this from a similar perspective.

```powershell
📁 Api/
|__ 📁 Apartments
    |__ 📁 ReserveApartment
    |__ ...
|__ 📁 Bookings
    |__ 📁 CancelBooking
    |__ ...
|__ 📁 Payments
|__ 📁 Reviews
|__ 📁 Disputes
|__ 📁 Invoicing
```

The use case driven folder structure helps us better understand user needs and aligns development efforts with business goals.

## Screaming Architecture Benefits

The benefits of organizing our system around use cases are:

- Improved cohesion since related use cases are close together
- High coupling for a single use case and its related use cases
- Low coupling between unrelated use cases
- Easier navigation through the solution

## Bounded Contexts and Vertical Slices

We have many techniques for discovering the high-level modules within our system.
For example, we could use [event storming](https://www.eventstorming.com/) to explore the system's use cases.
Domain exploration happens before we write a single line of code.

The next step is decomposing the larger problem domain into smaller sub-domains and later bounded contexts.
This gives us loosely coupled high-level modules that we can translate into code.

<Image
  src="blogs/mnw_104/bounded_contexts.png"
  width={1060}
  height={187}
  alt="Bounded contexts."
/>

The overarching idea here is thinking about cohesion around functionalities.
We want to organize our system so that the cohesion between the components is high.
Bounded contexts, vertical slices, and screaming architecture are complementary concepts.

Here's a screaming architecture example for this system.
Let's say the `Ticketing` module uses [**Clean Architecture**](clean-architecture-folder-structure) internally.
But we can still organize the system around feature folders and use cases.
An alternative approach could be organizing around [**vertical slices**](vertical-slice-architecture-structuring-vertical-slices), resulting in a less nested folder structure.

```powershell
📁 Modules/
|__ 📁 Attendance
    |__ ...
|__ 📁 Events
    |__ ...
|__ 📁 Ticketing
    |__ 📁 Application
        |__ 📁 Carts
            |__ 📁 AddItemToCart
            |__ 📁 ClearCart
            |__ 📁 GetCart
            |__ 📁 RemoveItemFromCart
        |__ 📁 Orders
            |__ 📁 SubmitOrder
            |__ 📁 CancelOrder
            |__ 📁 GetOrder
        |__ 📁 Payments
            |__ 📁 RefundPayment
        |__ ...
    |__ 📁 Domain
        |__ 📁 Customers
        |__ 📁 Orders
        |__ 📁 Payments
        |__ 📁 Tickets
        |__ ...
    |__ 📁 infrastructure
        |__ 📁 Authentication
        |__ 📁 Customers
        |__ 📁 Database
        |__ 📁 Orders
        |__ 📁 Payments
        |__ 📁 Tickets
        |__ ...
|__ 📁 Users
    |__ ...
```

The example above is a small part of the system I built inside of [**Modular Monolith Architecture**](/modular-monolith-architecture).

## Takeaway

**Screaming Architecture** isn't just a catchy phrase, it's an approach that can profoundly impact how you build software.
By organizing your system around use cases, you align your codebase with the core business domain.
Your system exists to solve the business domain problems.

Remember, the goal is to create a system that communicates its purpose through its structure.
Embrace a use case-driven approach, break down complex domains into bounded contexts.
Build a system that truly "screams" about the problems it solves.

If you want to explore these powerful ideas further, check out [**Pragmatic Clean Architecture**](/pragmatic-clean-architecture).
I share my entire framework for building robust applications from the ground up and organizing the system around use cases.

That's all for today.

See you next week.

---



<Sponsorships>

Introducing [**IronSecureDoc**](https://ironsoftware.com/enterprise/securedoc/?utm_source=milan&utm_medium=email&utm_campaign=milan) -
the PDF security & compliance server.
Protect your PDFs with IronSecureDoc, the latest from Iron Software.
Redact sensitive data, add digital signatures, and stay compliant with PDF/A & PDF/UA standards.
Easy integration with our REST API.
[**Learn more here**](https://ironsoftware.com/enterprise/securedoc/?utm_source=milan&utm_medium=email&utm_campaign=milan).

Test your API using the [**Collection Runner**](https://learning.postman.com/docs/collections/running-collections/intro-to-collection-runs/).
Use Collection Runner to log the test results for your requests, and use scripts to pass data between requests and change the request workflow.
[**Check it out here**](https://learning.postman.com/docs/collections/running-collections/intro-to-collection-runs/).

</Sponsorships>

Have you ever wondered how large-scale systems handle traffic spikes or maintain performance even when parts of the system are temporarily down?
The answer lies in asynchronous messaging.

Asynchronous messaging is, at its core, about decoupling.
Our components can operate independently and communicate through a message queue or topic.
If one service (component) is temporarily unavailable, the others can continue working.
This improves our system's scalability, resilience, and fault tolerance.

In this article, we'll explore how to use Amazon SQS and SNS for asynchronous messaging in .NET applications.

We'll also see how MassTransit simplifies the process, enabling you to build robust message-driven systems.

Let's dive in.

## What is Amazon SQS?

[Amazon Simple Queue Service](https://aws.amazon.com/sqs/) (SQS) is a fully managed message queueing service.
It facilitates the decoupling and scaling of microservices and distributed systems.

SQS acts as a reliable middleman for asynchronous communication.
It enables different components of your architecture to exchange messages without needing to be online or directly connected at the same time.
Messages are stored in queues and consumed on demand.

SQS offers two distinct queue types depending on your requirements:

- **Standard Queues**: Ideal for high-throughput scenarios.
  Standard Queues provide at-least-once delivery and best-effort ordering.
- **FIFO Queues**: Recommended when maintaining message order is required.
  FIFO Queues guarantee exactly-once processing and preserve the sequence in which messages are sent.

Let's say we have two services - `Stock` and `Reporting`.
When a user creates a purchase order in the `Stock` service, we want to notify the `Reporting` service.

SQS allows us to create decoupled communication between these services.
The `Stock` service sends a message to an SQS queue, and the `Reporting` service can poll from the queue to consume messages.

<Image
  src="blogs/mnw_103/amazon_sqs.png"
  width={1920}
  height={1080}
  alt="Amazon SQS."
/>

It's interesting to highlight that SQS uses a polling mechanism for message consumers.
When a consumer polls for new messages, SQS starts a [visibility timeout](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html).
SQS doesn't automatically delete the messages.
Instead, messages are hidden from other consumers until the timeout expires.

When the consumer successfully processes a message, it's removed from the queue.
But if the visibility timeout expires, the message becomes visible and can be delivered again.
Other consumers can receive this message when polling from SQS.
This is why SQS offers at-least-once delivery (for standard queues).
You will have to implement [**idempotency in the consumer**](idempotent-consumer-handling-duplicate-messages).

## Amazon SQS and Competing Consumers

Let's introduce another service into our system - the `Risk Management` service.
When multiple consumers (services) poll an SQS queue, each wants to retrieve and process messages as they become available.
However, once a message is successfully received and processed by one consumer, it's removed from the queue.

Why is this a problem?

Other consumers who might have been polling will miss out on that specific message.
This is known as [competing consumers](https://learn.microsoft.com/en-us/azure/architecture/patterns/competing-consumers).

Let's consider the example of `Reporting` and `Risk Management` services polling the same queue.
If a new message arrives, only one of these services will "win" the race and retrieve it for processing.
The other service won't find that message even if it polls moments later.

<Image
  src="blogs/mnw_103/amazon_sqs_competing_consumers.png"
  width={1920}
  height={1080}
  alt="Amazon SQS and competing consumers."
/>

So, how can we solve this?

We could introduce a dedicated queue for each service.
However, the producer now needs to publish to multiple queues.
This creates a possibility for some (not so) interesting partial failures.
What happens if we successfully publish to one queue but fail to publish to the other?

You can see how this becomes difficult to scale while maintaining reliability.

Luckily, there's a solution.

## Amazon SNS to The Rescue

[Amazon Simple Notification Service](https://aws.amazon.com/sns/) (SNS) is a fully managed pub/sub messaging service.
It allows publishers to send messages to multiple subscribers (topics) simultaneously.

SNS operates on the principle of publishers and subscribers.
Publishers send messages to an SNS topic, while subscribers express interest in specific topics and receive messages published to those topics.
This decoupled architecture allows you to add or remove subscribers without impacting the publisher or other subscribers.

SNS seamlessly integrates with SQS, allowing you to create a powerful combination where SNS handles the fan-out of messages,
and SQS queues ensure that each message is processed exclusively by a single consumer (service).

Instead of sending a message to the queue, the `Stock` service now publishes to an SNS topic.
Both the `Reporting` and `Risk Management` services create their own SQS queues and subscribe these queues to the SNS topic.
When a new message is published to the SNS topic, SNS delivers it to both SQS queues.
Each queue receives its own copy of the message.

If we want to introduce a new service, we'll create a new SQS queue and subscribe it to the topic.

<Image
  src="blogs/mnw_103/amazon_sns.png"
  width={1920}
  height={1080}
  alt="Amazon SNS."
/>

## MassTransit Integration With SQS and SNS

How can we use SNS and SQS from a .NET application?

You could use the official AWS SDKs.
The benefit is you'll have more control over messaging.
However, you will need to write more code to receive and handle messages successfully.

So, I want to suggest a different approach.

MassTransit is one of the most popular messaging libraries in .NET.
It provides a set of messaging abstractions on top of the supported message transports.

I wrote an article about [**using MassTransit with RabbitMQ and Azure Service Bus**](using-masstransit-with-rabbitmq-and-azure-service-bus).

But we'll focus on using MassTransit with SQS and SNS.

Let's start by installing the NuGet package we'll need:

```powershell
Install-Package MassTransit.AmazonSQS
```

Next, we'll need to configure MassTransit with our .NET applications.

Here's the MassTransit configuration for the `Stock` service:

```csharp
builder.Services.AddMassTransit(configure =>
{
    configure.AddConsumer<PurchaseOrderSentConsumer>().Endpoint(e => e.InstanceId = "stocks");

    configure.UsingAmazonSqs((context, cfg) =>
    {
        cfg.Host("eu-central-1", h =>
        {
            h.AccessKey(builder.Configuration["AmazonSqs:AccessKey"]!);
            h.SecretKey(builder.Configuration["AmazonSqs:SecretKey"]!);

            h.Scope("stocks-platform", scopeTopics: true);
        });

        cfg.ConfigureEndpoints(context, new KebabCaseEndpointNameFormatter("stocks-platform-", false));
    });
});
```

Here are a few things I want to highlight here:

- `UsingAmazonSqs` - Configures SQS (and SNS) as the message transport.
- `Scope` - Adds a prefix to SNS topic names. This helps distinguish topics from other applications and environments.
- `Endpoint` - Sets the `InstanceId`, which is appended to the endpoint (queue) name.
- `ConfigureEndpoints` - Allows us to specify a prefix for endpoint (queue) names. The idea is the same as the topic name prefix.

You'll also need to configure an [IAM policy](https://masstransit.io/documentation/configuration/transports/amazon-sqs#example-iam-policy)
that gives MassTransit the required permissions for AWS resources.

And with this setup in place, you can use MassTransit to publish messages.
Here's an endpoint that accepts a purchase order and publishes a `PurchaseOrderSent` message.

```csharp
app.MapPost("purchase-orders", async (PurchaseOrderRequest request, IPublishEndpoint publishEndpoint) =>
{
    var purchaseOrder = new PurchaseOrder
    {
        Id = Guid.NewGuid(),
        Ticker = request.Ticker,
        LimitPrice = request.LimitPrice,
        Quantity = request.Quantity
    };

    OrdersDb.Add(purchaseOrder);

    await publishEndpoint.Publish(new PurchaseOrderSent(purchaseOrder.Id));

    return Results.Ok(purchaseOrder);
});
```

We will process the purchase order in the `PurchaseOrderSentConsumer`.
If it's successfully processed (filled), we will publish an `OrderFilled` message.

The `Risk Management` service can subscribe to this message using MassTransit.
The configuration is almost identical, with the only difference being the endpoint's `InstanceId`.

```csharp
builder.Services.AddMassTransit(configure =>
{
    configure.AddConsumer<OrderFilledConsumer>().Endpoint(e => e.InstanceId = "risk-management");

    configure.UsingAmazonSqs((context, cfg) =>
    {
        cfg.Host("eu-central-1", h =>
        {
            h.AccessKey(builder.Configuration["AmazonSqs:AccessKey"]!);
            h.SecretKey(builder.Configuration["AmazonSqs:SecretKey"]!);

            h.Scope("stocks-platform", true);
        });

        cfg.ConfigureEndpoints(context, new KebabCaseEndpointNameFormatter("stocks-platform-", false));
    });
});
```

## Broker Topology in AWS

MassTransit will automatically create the required queues, topics, and subscriptions in AWS.
If needed, you can further configure the SQS and SNS resources.

Of course, you can also create the required infrastructure in AWS and tell MassTransit to use it.

But let's keep it simple and allow MassTransit to provision the AWS resources.

By default, MassTransit creates standard SQS queues.
Here's what we get in Amazon SQS.

<div className="bordered">
  <Image
    src="blogs/mnw_103/amazon_sqs_queues.png"
    width={1070}
    height={720}
    alt="Amazon SQS queues created by MassTransit."
  />
</div>

And here's what we get in Amazon SNS:

<div className="bordered">
  <Image
    src="blogs/mnw_103/amazon_sns_topics.png"
    width={1280}
    height={650}
    alt="Amazon SNS topics created by MassTransit."
  />
</div>

MassTransit automatically configures the required subscriptions between the topic and queues.

And we are ready to start publishing and consuming messages.

## In Summary

Asynchronous communication and decoupling are pivotal in achieving scalability, resilience, and fault tolerance.
Amazon SQS and SNS provide the building blocks of message-driven architectures in the AWS cloud.

We explored the core concepts of SQS and SNS, understanding how they enable reliable message delivery and fan-out capabilities.

MassTransit provides an excellent abstraction layer over SQS and SNS, simplifying development.
We can focus on solving the business problems and delivering value to our users.

The combination of Amazon SQS, SNS, and MassTransit gives us robust tools for building modern, event-driven applications.

Thanks for reading, and I'll see you next week!

**P.S.** You can find the source code for this article in [**this repository**](https://github.com/m-jovanovic/aws-tutorials), under the `Amazon SQS and SNS` folder.

---



<Sponsorships>

[**WPF or .NET MAUI - How to Choose**](https://www.telerik.com/blogs/wpf-net-maui-how-choose?utm_medium=cpm&utm_source=milanjovanovic&utm_campaign=maui-general-awareness)?
Creating a desktop application and need to decide between the long-time reliable WPF and the up-and-coming cross-platform .NET MAUI?
[**This blog post**](https://www.telerik.com/blogs/wpf-net-maui-how-choose?utm_medium=cpm&utm_source=milanjovanovic&utm_campaign=maui-general-awareness)
will help you explore the features that matter.

[**Postman v11**](https://blog.postman.com/announcing-postman-v11/) takes your dev collaboration to next level using AI and APIs.
Check out [**Postman v11**](https://blog.postman.com/announcing-postman-v11/) to get started with it's features designed to help bring API collaboration to the next level.

</Sponsorships>

Okay, let's be honest.
We all have a million things on our plates, and diving deep into every nook and cranny of EF Core might not be
at the top of your priority list.

But here's the deal: EF Core is powerful, and knowing a few key features can save you lots of time and frustration.

So, I won't bombard you with every single EF Core feature under the sun.

Instead, I've cherry-picked five essential ones that you really need to know.

We'll go through:

- **Query Splitting** - your database's new best friend
- **Bulk Updates and Deletes** - efficiency on steroids
- **Raw SQL Queries** - when you need to go rogue
- **Query Filters** - keeping things nice and tidy
- **Eager Loading** - because lazy isn't so great

Let's get started!

## Query Splitting

[**Query splitting**](how-to-improve-performance-with-ef-core-query-splitting) is one of those EF Core features that you rarely need.
Until one day, you do.
Query splitting is helpful in scenarios where you're eager loading multiple collections.
It helps us avoid the [cartesian explosion](https://learn.microsoft.com/en-us/ef/core/performance/efficient-querying#avoid-cartesian-explosion-when-loading-related-entities) problem.

Let's say we want to retrieve a department with all its teams and employees.
We might write a query like this:

```csharp
Department department =
    context.Departments
        .Include(d => d.Teams)
        .Include(d => d.Employees)
        .Where(d => d.Id == departmentId)
        .First();
```

This translates to a single SQL query with two JOINs.
However, since these `JOIN` statements are on the same level, the database will return a _cross product_.
Each row from `Teams` will be joined with each row `Employees`.
In that case, the database returns many rows, significantly impacting performance.

Here's how we can avoid these performance issues with query splitting:

```csharp
Department department =
    context.Departments
        .Include(d => d.Teams)
        .Include(d => d.Employees)
        .Where(d => d.Id == departmentId)
        .AsSplitQuery()
        .First();
```

With `AsSplitQuery`, EF Core will execute an additional SQL query for each collection navigation.

However, be cautious not to overuse query splitting.
I use split queries when I've _measured_ that they consistently perform better.

Split queries have more round trips to the database, which might be slower if database latency is high.
There is also no consistency guarantee across multiple SQL queries.

## Bulk Updates and Deletes

EF Core 7 added two new APIs for performing [**bulk updates and deletes**](how-to-use-the-new-bulk-update-feature-in-ef-core-7),
`ExecuteUpdate` and `ExecuteDelete`.
They allow you to efficiently update a large number of rows in one round trip to the database.

Here's a practical example.

The company has decided to give a 5% raise to all employees in the "Sales" department.
Without bulk updates, we might iterate through each employee and update their salary individually:

```csharp
var salesEmployees = context.Employees
    .Where(e => e.Department == "Sales")
    .ToList();

foreach (var employee in salesEmployees)
{
    employee.Salary *= 1.05m;
}

context.SaveChanges();
```

This approach involves multiple database roundtrips, which can be inefficient, especially for large datasets.

We can achieve the same in one roundtrip using `ExecuteUpdate`:

```csharp
context.Employees
    .Where(e => e.Department == "Sales")
    .ExecuteUpdate(s => s.SetProperty(e => e.Salary, e => e.Salary * 1.05m));
```

This executes a single SQL `UPDATE` statement, directly modifying the salaries in the database without loading entities into memory, giving us improved performance.

Here's another example.
Let's say an e-commerce platform wants to delete all shopping carts older than one year.

Here's how we could do this with `ExecuteDelete`:

```csharp
context.Carts
    .Where(o => o.CreatedOn < DateTime.Now.AddYears(-1))
    .ExecuteDelete();
```

This results in a single SQL `DELETE` statement, directly removing the old shopping carts from the database.

However, bulk updates bypass the EF change tracker.
This could be problematic, and I wrote about the [**caveats of bulk updates in this article**](what-you-need-to-know-about-ef-core-bulk-updates).

## Raw SQL Queries

EF Core 8 added a new feature that allows us to query unmapped types with raw SQL.

Suppose we want to retrieve data from a database view, stored procedure, or a table that doesn't directly correspond to any of our entity classes.

For example, we want to retrieve a sales summary for each product.
With EF Core 8, we can define a simple `ProductSummary` class representing the structure of the result set and query it directly:

```csharp
public class ProductSummary
{
    public int ProductId { get; set; }
    public string ProductName { get; set; }
    public decimal TotalSales { get; set; }
}

var productSummaries = await context.Database
    .SqlQuery<ProductSummary>(
        @$"""
        SELECT p.ProductId, p.ProductName, SUM(oi.Quantity * oi.UnitPrice) AS TotalSales
        FROM Products p
        JOIN OrderItems oi ON p.ProductId = oi.ProductId
        WHERE p.CategoryId = {categoryId}
        GROUP BY p.ProductId, p.ProductName
        """)
    .ToListAsync();
```

The `SqlQuery` method returns an `IQueryable`, which allows you to compose raw SQL queries with LINQ.
This combines the power of raw SQL with the expressiveness of LINQ.

Remember to use parameterized queries to prevent SQL injection vulnerabilities.
The `SqlQuery` method accepts a `FormattableString`, which means you can safely use an interpolated string.
Each argument is converted to a SQL parameter.

You can learn more about [**raw SQL queries in this article**](ef-core-raw-sql-queries).

## Query Filters

[**Query filters**](how-to-use-global-query-filters-in-ef-core) are like reusable `WHERE` clauses you can apply to your entities.
These filters are automatically added to LINQ queries whenever you retrieve entities of the corresponding type.
This saves you from repeatedly writing the same filtering logic in multiple places within your application.

Query Filters are commonly used for scenarios like:

- [**Soft Deletes**](implementing-soft-delete-with-ef-core): Filter out records marked as deleted.
- [**Multi-tenancy**](multi-tenant-applications-with-ef-core): Filter data based on the current tenant.
- Row-level security: Restrict access to certain records based on user roles or permissions.

In a multi-tenant application, you often need to filter data based on the current tenant.
Query filters allow us to handle this requirement easily:

```csharp
public class Product
{
    public int Id { get; set; }
    public string Name { get; set; }
    // Associate products with tenants
    public int TenantId { get; set; }
}

protected override void OnModelCreating(ModelBuilder modelBuilder)
{
    // The current TenantId is set based on the current request/context
    modelBuilder.Entity<Product>().HasQueryFilter(p => p.TenantId == _currentTenantId);
}

// Now, queries automatically filter based on the tenant:
var productsForCurrentTenant = context.Products.ToList();
```

Configuring multiple query filters on the same entity will only apply the last one.
You can combine multiple query filters using `&&` (AND) and `||` (OR) operators.

You can use `IgnoreQueryFilters` to bypass the filters in specific queries when needed.

## Eager Loading

Eager Loading is a feature in EF Core that allows you to load related entities along with your main entity in a single database query.
By fetching all necessary data in a single query, you can improve application performance.
This is especially true when dealing with complex object graphs or when lazy loading would result in many small, inefficient queries.

Here's an example `VerifyEmail` use case.
We want to load an `EmailVerificationToken` and eagerly load a `User` with the `Include` method because we want to modify both entities at the same time.

```csharp {6}
internal sealed class VerifyEmail(AppDbContext context)
{
    public async Task<bool> Handle(Guid tokenId)
    {
        EmailVerificationToken? token = await context.EmailVerificationTokens
            .Include(e => e.User)
            .FirstOrDefaultAsync(e => e.Id == tokenId);

        if (token is null || token.ExpiresOnUtc < DateTime.UtcNow || token.User.EmailVerified)
        {
            return false;
        }

        token.User.EmailVerified = true;

        context.EmailVerificationTokens.Remove(token);

        await context.SaveChangesAsync();

        return true;
    }
}
```

EF Core will generate a single SQL query that joins the `EmailVerificationToken` and `User` tables, retrieving all the necessary data in one go.

Eager loading (and query splitting, which we mentioned earlier) isn't a silver bullet.
Consider using projections if you only need specific properties from related entities to avoid fetching unnecessary data.

## Summary

So, there you have it!
Five EF Core features that, frankly, you can't afford _not_ to know.
Remember, mastering EF Core takes time, but these features provide a solid foundation to build upon.

Another piece of advice is to deeply understand how your database works.
Mastering SQL also allows you to get the most value from EF Core.

While we focused on five key features, there are many other EF Core features worth exploring:

- [**Optimistic concurrency control**](solving-race-conditions-with-ef-core-optimistic-locking)
- [**Database migrations**](efcore-migrations-a-detailed-guide)
- [**Compiled queries**](unleash-ef-core-performance-with-compiled-queries)
- [**Transactions**](working-with-transactions-in-ef-core)
- [**Interceptors**](how-to-use-ef-core-interceptors)

EF Core is continuously evolving, so keep an eye on the latest updates and releases to stay ahead.

Good luck out there, and see you next week.

---


Previous issues of The .NET Weekly