Most RAG tutorials hand you the same shape of code.

Load some content. Chunk it. Embed it. Query it.

It works for a demo, but the moment you try to run that workflow repeatedly across different documentation sites, retrieval quality becomes the real problem -not embeddings, not vector databases, not orchestration frameworks.

Debugging retrieval is where the time goes.

You change a content selector. Re-ingest. Query again.

The chunking is wrong.  Fix it. Re-ingest. Query again.

The crawler pulled navigation instead of the article body.  Fix it. Re-ingest. Query again.

Every iteration takes longer than it should because the infrastructure sits in the middle of the feedback loop.

What I wanted was not another production-ready RAG platform.  I wanted a workbench.

Something that would let me:

• Crawl a documentation site
• Run end-to-end ingest locally
• Inspect chunks and retrieval scores
• Iterate on parser and chunking config quickly
• Debug retrieval failures without fighting infrastructure
• Carry the working configuration into production later

That became DocIngestion.  Built in .NET using Claude Code, it’s a local-first RAG ingestion and retrieval inspection tool designed around one thing:

• shortening the retrieval debugging loop.

Lets dig in.

~

The Architectural Decision That Changed Everything

The most important decision in the entire tool was also the simplest:  a single IVectorStore abstraction with 3 implementations.

public interface IVectorStore
{
    Task UpsertAsync(
        string indexName,
        IReadOnlyList<Chunk> chunks,
        CancellationToken ct);

    Task<IReadOnlyList<ScoredChunk>> QueryAsync(
        string indexName,
        float[] queryEmbedding,
        int topK,
        CancellationToken ct);

    Task<IReadOnlyList<IndexedDocument>> ListDocumentsAsync(
        string indexName,
        CancellationToken ct);
}

That interface sits underneath the ingest pipeline:

The pipeline never knows where the chunks are stored.

There are only two implementations:

• JSON on disk for iteration
• Elasticsearch for production parity

That’s it.

For most early-stage retrieval work, JSON is the better tool.  Not because it’s scalable, but because it’s quick, easy to eyeball, and gives you something good enough to iterate on during those crucial early stages.

You can optimise for scale and production later, once you’ve nailed down retrieval quality and pipeline behaviour.

~

Why JSON Beats a Vector Database Early On

When most people start a RAG project, they immediately provision infrastructure.

Pinecone. Qdrant. Elasticsearch. Azure AI Search.  That makes sense once the retrieval behaviour is already understood.

Before that, it’s mostly friction.  For a few thousand chunks, a JSON file is more than enough.

The implementation is deliberately primitive:

public sealed class JsonFileVectorStore : IVectorStore
{
    public async Task<IReadOnlyList<ScoredChunk>> QueryAsync(
        string indexName,
        float[] queryEmbedding,
        int topK,
        CancellationToken ct)
    {
        var chunks = await LoadAsync(indexName, ct);

        return chunks
            .Select(c => new ScoredChunk(
                c,
                CosineSimilarity(queryEmbedding, c.Embedding)))
            .OrderByDescending(s => s.Score)
            .Take(topK)
            .ToList();
    }
}

No infrastructure.  No provisioning.  No auth.  No cluster configuration.

Just:

• crawl
• ingest
• inspect
• query
• rerun

That changes the speed of iteration completely.  A broken content selector stops being a deployment problem and becomes a two-minute fix.

~

Retrieval Debugging Is the Real Problem

The most useful part of the workbench is not ingestion.  It’s inspection.

When retrieval quality is poor, there are usually only three questions:

1. Is the correct chunk in the index?
2. Is it being retrieved?
3. Is it ranking highly enough?

The workbench surfaces all three immediately.  Query a tenant and the UI shows:

• the synthesized answer
• retrieved chunks
• similarity scores
• source URLs
• indexed page counts

That turns retrieval debugging into a fast feedback loop instead of an infrastructure exercise.

If the chunk is missing entirely, the crawler or parser is wrong.

If the chunk exists but never retrieves, the embedding or query phrasing is wrong.

If the chunk retrieves but ranks badly, the chunking strategy is usually the issue.

That sounds obvious written down.  In practice, most RAG tooling often makes answering those questions slow.

~

The Bugs This Catches Quickly

The inspection workflow caught issues that would have taken hours to diagnose inside a cluster:

• A content selector scoped to the navigation panel instead of the article body, producing near-identical chunks across pages
• A chunker splitting code samples mid-function so signatures and implementations were separated
• A crawl indexing CDN error pages as canonical content across dozens of URLs

All of those became obvious by looking at:

• chunk counts
• retrieval scores
• duplicate URLs
• chunk text directly on disk

That’s the part most RAG tutorials skip.  Retrieval problems are usually data problems.

The faster you can inspect the data, the faster you improve retrieval.

~

Elasticsearch Still Matters

The point is not that JSON replaces a vector database.  It doesn’t.  The JSON store is intentionally limited:

• everything loads into memory
• concurrent writes are unsafe
• performance drops as chunk counts grow

The point is that JSON is a better environment for discovering what “good retrieval” actually looks like.  Once retrieval quality is understood, switching to Elasticsearch is a config change:

"storage": {
  "type": "elasticsearch"
}

The pipeline code stays the same.

That’s where the abstraction earns its keep.

Whatever I learn while iterating locally, parser rules, chunk sizes, overlap strategy,  or retrieval tuning, it all carries directly into the production implementation.  No rewrite required.

No migration project.  No second ingest pipeline.

~

The Most Useful Part

The most useful abstractions are the ones that let you defer decisions until you have enough evidence to make them properly.

IVectorStore is not a complicated abstraction.  It’s three methods.

What it gave me was a way to:

• debug retrieval quickly
• iterate locally
• inspect data directly
• avoid infrastructure too early
• move into Elasticsearch later without changing the pipeline

Most RAG problems are not solved by adding more infrastructure.

They’re solved by shortening the feedback loop between ingest and retrieval quality.

That’s what this workbench was built for.

~

Enjoy what you’ve read, have questions about this content, or would like to see another topic?

You can schedule a call using my Calendly link to discuss consulting and development services.

~

Courses

Check my AI courses.  From developers to decision makers, these have you covered:

• Developing an Artificial Intelligence Strategy for Your Organization
• Aligning Generative AI with Business Cases
• Vector Databases & Embeddings for Developers

~