Building a RAG System in .NET

To build a RAG system in .NET, you index documents by chunking them, embedding each chunk, and storing the vectors, then answer queries by retrieving the most similar chunks and passing them to the model as context. Microsoft.Extensions.AI handles the models, and PostgreSQL with pgvector handles storage.

A demo takes an afternoon; a system that reliably answers from your own documents needs chunking that preserves context and retrieval that actually finds the right chunks. Let's build that full pipeline.

What Is RAG?

Retrieval-Augmented Generation (RAG) is a pattern that combines information retrieval with AI text generation. Instead of relying solely on what the model learned during training, RAG retrieves relevant documents from your data and includes them in the prompt. This grounds the AI's response in your actual content.

Without retrieval, a model answering questions about your company's policies has no access to the current source of truth. RAG retrieves relevant sections and places them in the prompt, but retrieval can still miss and the model can still produce an unsupported answer. Source citations and an explicit "insufficient context" path are part of the design, not optional polish.

The RAG Pipeline

A RAG system has two main phases:

1. Indexing: Convert your documents into vector embeddings and store them
2. Querying: Convert the user's question into an embedding, find similar documents, and generate a response

Indexing runs ahead of time, turning each document into searchable vectors:

Querying happens at request time, retrieving relevant chunks and passing them to the model as context:

// IVectorStore and DocumentChunker here are our own types (defined below),
// not the abstractions from Microsoft.Extensions.VectorData
public class RagPipeline
{
    private readonly IEmbeddingGenerator<string, Embedding<float>> _embedder;
    private readonly IChatClient _chatClient;
    private readonly IVectorStore _vectorStore;
    private readonly DocumentChunker _chunker;

    public RagPipeline(
        IEmbeddingGenerator<string, Embedding<float>> embedder,
        IChatClient chatClient,
        IVectorStore vectorStore,
        DocumentChunker chunker)
    {
        _embedder = embedder;
        _chatClient = chatClient;
        _vectorStore = vectorStore;
        _chunker = chunker;
    }

    public async Task IndexDocumentAsync(Document document)
    {
        var chunks = _chunker.ChunkText(document.Content, document.FileName);

        foreach (var chunk in chunks)
        {
            ReadOnlyMemory<float> vector = await _embedder.GenerateVectorAsync(chunk.Text);

            await _vectorStore.UpsertAsync(new VectorEntry
            {
                Id = chunk.Id,
                Text = chunk.Text,
                Embedding = vector.ToArray(),
                Metadata = new Dictionary<string, string>
                {
                    ["source"] = document.FileName,
                    ["chunkIndex"] = chunk.ChunkIndex.ToString()
                }
            });
        }
    }

    public async Task<string> QueryAsync(string question)
    {
        // Step 1: Generate embedding for the question
        ReadOnlyMemory<float> queryVector = await _embedder
            .GenerateVectorAsync(question);

        // Step 2: Find relevant documents
        var relevantDocs = await _vectorStore.SearchAsync(
            queryVector.ToArray(),
            topK: 5);

        // Step 3: Generate response with context
        var context = string.Join("\n\n",
            relevantDocs.Select(d => d.Text));

        var messages = new List<ChatMessage>
        {
            new(ChatRole.System, $"""
                Answer the user's question based ONLY on the following context.
                If the answer is not in the context, say "I don't have information
                about that in my knowledge base."

                Context:
                {context}
                """),
            new(ChatRole.User, question)
        };

        var response = await _chatClient.GetResponseAsync(messages);
        return response.Text;
    }
}

Document Chunking

Large documents must be split into smaller chunks for effective retrieval. Chunk size affects both retrieval quality and cost.

Two small types carry the data through the pipeline:

public record Document(string FileName, string Content);

public class DocumentChunk
{
    public required string Id { get; init; }
    public required string Text { get; init; }
    public required string SourceId { get; init; }
    public required int ChunkIndex { get; init; }
}

And the chunker itself:

public class DocumentChunker
{
    private readonly int _chunkSize;
    private readonly int _chunkOverlap;

    public DocumentChunker(int chunkSize = 2000, int chunkOverlap = 200)
    {
        _chunkSize = chunkSize;
        _chunkOverlap = chunkOverlap;
    }

    public List<DocumentChunk> ChunkText(string text, string sourceId)
    {
        var chunks = new List<DocumentChunk>();
        var sentences = SplitIntoSentences(text);

        var currentChunk = new StringBuilder();
        var chunkIndex = 0;
        var overlapBuffer = new Queue<string>();

        foreach (var sentence in sentences)
        {
            if (currentChunk.Length + sentence.Length > _chunkSize
                && currentChunk.Length > 0)
            {
                chunks.Add(new DocumentChunk
                {
                    Id = $"{sourceId}-chunk-{chunkIndex}",
                    Text = currentChunk.ToString().Trim(),
                    SourceId = sourceId,
                    ChunkIndex = chunkIndex
                });

                chunkIndex++;

                // Keep overlap text for the next chunk
                currentChunk.Clear();
                foreach (var overlapSentence in overlapBuffer)
                {
                    currentChunk.Append(overlapSentence).Append(' ');
                }
            }

            currentChunk.Append(sentence).Append(' ');

            overlapBuffer.Enqueue(sentence);
            while (string.Join(" ", overlapBuffer).Length > _chunkOverlap)
            {
                overlapBuffer.Dequeue();
            }
        }

        if (currentChunk.Length > 0)
        {
            chunks.Add(new DocumentChunk
            {
                Id = $"{sourceId}-chunk-{chunkIndex}",
                Text = currentChunk.ToString().Trim(),
                SourceId = sourceId,
                ChunkIndex = chunkIndex
            });
        }

        return chunks;
    }

    private static string[] SplitIntoSentences(string text)
    {
        return text.Split(
            new[] { ". ", "! ", "? ", "\n\n" },
            StringSplitOptions.RemoveEmptyEntries);
    }
}

The overlap ensures that information spanning chunk boundaries isn't lost. The chunker counts characters as a cheap proxy for tokens (one token is roughly 4 characters). The 2,000-character default lands near 500 tokens with a 10% overlap, which works well for most content.

Vector Storage With PostgreSQL

Using pgvector for vector storage. If you're weighing pgvector against dedicated options like Qdrant, I compared them in vector databases for .NET applications.

Install the packages:

dotnet add package Npgsql
dotnet add package Pgvector

The IVectorStore abstraction keeps the pipeline decoupled from the storage engine:

public interface IVectorStore
{
    Task UpsertAsync(VectorEntry entry);
    Task<List<VectorSearchResult>> SearchAsync(float[] queryEmbedding, int topK = 5);
}

public class VectorEntry
{
    public required string Id { get; init; }
    public required string Text { get; init; }
    public required float[] Embedding { get; init; }
    public required Dictionary<string, string> Metadata { get; init; }
}

public class VectorSearchResult
{
    public required string Id { get; init; }
    public required string Text { get; init; }
    public required Dictionary<string, string> Metadata { get; init; }
    public required double Similarity { get; init; }
}

And the pgvector implementation:

public class PgVectorStore : IVectorStore
{
    private readonly NpgsqlDataSource _dataSource;

    public PgVectorStore(NpgsqlDataSource dataSource)
    {
        _dataSource = dataSource;
    }

    public async Task InitializeAsync()
    {
        await using var cmd = _dataSource.CreateCommand("""
            CREATE EXTENSION IF NOT EXISTS vector;

            CREATE TABLE IF NOT EXISTS document_embeddings (
                id TEXT PRIMARY KEY,
                text TEXT NOT NULL,
                embedding vector(1536),
                metadata JSONB,
                created_at TIMESTAMPTZ DEFAULT NOW()
            );

            CREATE INDEX IF NOT EXISTS idx_embeddings_ivfflat
                ON document_embeddings
                USING ivfflat (embedding vector_cosine_ops)
                WITH (lists = 100);
            """);

        await cmd.ExecuteNonQueryAsync();
    }

    public async Task UpsertAsync(VectorEntry entry)
    {
        await using var cmd = _dataSource.CreateCommand("""
            INSERT INTO document_embeddings (id, text, embedding, metadata)
            VALUES ($1, $2, $3::vector, $4::jsonb)
            ON CONFLICT (id) DO UPDATE SET
                text = EXCLUDED.text,
                embedding = EXCLUDED.embedding,
                metadata = EXCLUDED.metadata
            """);

        cmd.Parameters.AddWithValue(entry.Id);
        cmd.Parameters.AddWithValue(entry.Text);
        cmd.Parameters.AddWithValue(
            new Vector(entry.Embedding));
        cmd.Parameters.AddWithValue(
            JsonSerializer.Serialize(entry.Metadata));

        await cmd.ExecuteNonQueryAsync();
    }

    public async Task<List<VectorSearchResult>> SearchAsync(
        float[] queryEmbedding, int topK = 5)
    {
        await using var cmd = _dataSource.CreateCommand("""
            SELECT id, text, metadata,
                   1 - (embedding <=> $1::vector) as similarity
            FROM document_embeddings
            ORDER BY embedding <=> $1::vector
            LIMIT $2
            """);

        cmd.Parameters.AddWithValue(new Vector(queryEmbedding));
        cmd.Parameters.AddWithValue(topK);

        var results = new List<VectorSearchResult>();

        await using var reader = await cmd.ExecuteReaderAsync();
        while (await reader.ReadAsync())
        {
            results.Add(new VectorSearchResult
            {
                Id = reader.GetString(0),
                Text = reader.GetString(1),
                Metadata = JsonSerializer.Deserialize<Dictionary<string, string>>(
                    reader.GetString(2))!,
                Similarity = reader.GetDouble(3)
            });
        }

        return results;
    }
}

The Complete RAG API

Wire everything together in an ASP.NET Core API:

var builder = WebApplication.CreateBuilder(args);

// AI services
var openAiClient = new OpenAIClient(builder.Configuration["OpenAI:ApiKey"]);

builder.Services.AddChatClient(
    openAiClient.GetChatClient("gpt-4o").AsIChatClient());

builder.Services.AddEmbeddingGenerator(
    openAiClient.GetEmbeddingClient("text-embedding-3-small")
        .AsIEmbeddingGenerator());

// Npgsql data source with pgvector support.
// UseVector() (from the Pgvector.Npgsql namespace) teaches Npgsql
// to map Vector parameters. Without it, every Vector parameter throws.
var dataSourceBuilder = new NpgsqlDataSourceBuilder(
    builder.Configuration.GetConnectionString("Postgres"));
dataSourceBuilder.UseVector();
builder.Services.AddSingleton(dataSourceBuilder.Build());

// Vector store
builder.Services.AddSingleton<PgVectorStore>();
builder.Services.AddSingleton<IVectorStore>(
    sp => sp.GetRequiredService<PgVectorStore>());
builder.Services.AddSingleton<DocumentChunker>();
builder.Services.AddScoped<RagPipeline>();

var app = builder.Build();

// Create the vector extension, table, and index before serving traffic
await app.Services.GetRequiredService<PgVectorStore>().InitializeAsync();

// Index a document
app.MapPost("/api/documents", async (
    RagPipeline rag,
    DocumentUploadRequest request) =>
{
    await rag.IndexDocumentAsync(
        new Document(request.FileName, request.Content));

    return Results.Ok("Document indexed successfully");
});

// Query the knowledge base
app.MapPost("/api/query", async (
    RagPipeline rag,
    QueryRequest request) =>
{
    var answer = await rag.QueryAsync(request.Question);
    return Results.Ok(new { Answer = answer });
});

app.Run();

public record DocumentUploadRequest(string FileName, string Content);

public record QueryRequest(string Question);

The InitializeAsync call runs the schema setup on startup, so the extension, table, and index exist before the first request hits the API.

Improving Retrieval Quality

Basic RAG can be improved with several techniques:

Hybrid Search

Combine vector similarity with keyword search for better results:

public async Task<List<VectorSearchResult>> HybridSearchAsync(
    float[] queryEmbedding, string queryText, int topK = 5)
{
    await using var cmd = _dataSource.CreateCommand("""
        WITH vector_results AS (
            SELECT id, text, metadata,
                   1 - (embedding <=> $1::vector) as vector_score
            FROM document_embeddings
            ORDER BY embedding <=> $1::vector
            LIMIT $3
        ),
        text_results AS (
            SELECT id, text, metadata,
                   ts_rank(to_tsvector('english', text),
                           plainto_tsquery('english', $2)) as text_score
            FROM document_embeddings
            WHERE to_tsvector('english', text) @@
                  plainto_tsquery('english', $2)
            LIMIT $3
        )
        SELECT COALESCE(v.id, t.id) as id,
               COALESCE(v.text, t.text) as text,
               COALESCE(v.metadata, t.metadata) as metadata,
               COALESCE(v.vector_score, 0) * 0.7 +
               COALESCE(t.text_score, 0) * 0.3 as combined_score
        FROM vector_results v
        FULL OUTER JOIN text_results t ON v.id = t.id
        ORDER BY combined_score DESC
        LIMIT $3
        """);

    cmd.Parameters.AddWithValue(new Vector(queryEmbedding));
    cmd.Parameters.AddWithValue(queryText);
    cmd.Parameters.AddWithValue(topK);

    var results = new List<VectorSearchResult>();

    await using var reader = await cmd.ExecuteReaderAsync();
    while (await reader.ReadAsync())
    {
        results.Add(new VectorSearchResult
        {
            Id = reader.GetString(0),
            Text = reader.GetString(1),
            Metadata = JsonSerializer.Deserialize<Dictionary<string, string>>(
                reader.GetString(2))!,
            Similarity = reader.GetDouble(3)
        });
    }

    return results;
}

Re-Ranking

Score retrieved documents again for relevance after initial retrieval:

public class ReRanker
{
    private readonly IChatClient _chatClient;

    public ReRanker(IChatClient chatClient)
    {
        _chatClient = chatClient;
    }

    public async Task<List<VectorSearchResult>> ReRankAsync(
        string query,
        List<VectorSearchResult> candidates)
    {
        var prompt = $"""
            Given the query: "{query}"

            Rate each document's relevance from 0.0 to 1.0:
            {string.Join("\n", candidates.Select((c, i) =>
                $"Document {i}: {c.Text[..Math.Min(200, c.Text.Length)]}"))}

            Return a JSON array of scores, one per document.
            """;

        var response = await _chatClient.GetResponseAsync(prompt);
        var scores = JsonSerializer.Deserialize<double[]>(response.Text);

        return candidates
            .Zip(scores!, (doc, score) => (doc, score))
            .OrderByDescending(x => x.score)
            .Select(x => x.doc)
            .ToList();
    }
}

Source Attribution

Always show users where the information came from:

public class RagResponse
{
    public string Answer { get; set; } = string.Empty;
    public List<SourceReference> Sources { get; set; } = new();
}

public class SourceReference
{
    public string DocumentName { get; set; } = string.Empty;
    public int ChunkIndex { get; set; }
    public double Relevance { get; set; }
}

Include sources in the response so users can verify the information. This builds trust and helps identify when the RAG system needs better data.

When RAG Isn't the Answer

RAG is not the solution for every AI feature:

• Small, stable knowledge: if your entire knowledge base fits in a few thousand tokens, just put it in the system prompt. No pipeline needed.
• Structured data questions: "How many orders did we ship last month?" is a SQL query, not a retrieval problem. Consider function calling that lets the model query your APIs instead.
• Reasoning over the whole corpus: RAG retrieves the top few chunks. Questions like "summarize all customer complaints this year" need aggregation, not similarity search.

And if you only need search results (not generated answers), skip the generation step entirely and build semantic search instead. It's cheaper and faster.

Summary

The pipeline is the product: chunking, embeddings, and retrieval quality decide whether answers come from your data or from thin air.

1. RAG grounds AI responses in your actual data by retrieving relevant documents before generation
2. Document chunking with overlap ensures information spanning boundaries is preserved
3. PostgreSQL with pgvector provides a production-ready vector store without additional infrastructure
4. Hybrid search combining vector similarity and keyword matching improves retrieval quality
5. Re-ranking retrieved documents improves the relevance of context provided to the AI
6. Always include source attribution so users can verify the AI's responses
7. RAG is the foundation for knowledge-based AI features - chatbots, search, documentation assistants

Thanks for reading, and stay awesome!