Your RAG Pipeline Is Slow Because You’re Sending Entire Pages to the LLM

I recently debugged an MCP documentation search implementation running on .NET that was taking 18–25+ seconds per query.

Far too slow for interactive use.

The pipeline was straightforward: Elasticsearch KNN vector search, a reranker, then GPT-4.1-mini for synthesis.

The latency was making it unusable inside GitHub Copilot and VS Code.

Before touching any config or swapping the model, the right move was to add diagnostic logging and find out what was actually being sent to the LLM. What came back was instructive.

Let’s dig in.

~

The Setup

The RAG pipeline looked like this:

1. Incoming query gets embedded
2. Elasticsearch KNN search retrieves the top n chunks
3. A reranker orders them by relevance
4. The top chunks are passed to GPT-4.1-mini for synthesis
5. The generated answer is returned to the client

 

The code responsible for preparing chunks before synthesis lived in a method CleanChunks:

private IEnumerable<RagChunk> CleanChunks(IEnumerable<RagChunk> chunks)
{
    return chunks
        .Where(c => c.Score >= MinRelevanceScore)
        .OrderByDescending(c => c.Score)
        .Take(MaxChunksForSynthesis);
}

 

Clean and simple.  It turned out this was the source of the problem.

~

What the Logs Actually Showed

Diagnostic logging was added to capture chunk count, token count, and document URIs before the synthesis call. The output for a straightforward deployment query looked like this:

Chunks retrieved: 8
Top chunk score: 8.665282
Document URIs: deploying-wasm/, deploying-wasm/, deploying-wasm/,
               deploying-wasm/, deploying-wasm/, api-reference/,
               api-reference/, quickstart/
Token count sent to synthesis LLM: ~19,700

 

Five of the eight chunks came from the same document URI, each with an identical relevance score of 8.665282.

The KNN search was returning paragraph-level chunks from the same page, so the synthesis step was receiving the equivalent of an entire documentation page rather than a few focused, diverse results.

Nearly 20,000 tokens for a single query.

The model wasn’t slow. The chunk size wasn’t the problem. The pipeline was sending too much because there was no deduplication by document source.

~

The Fix: GroupBy DocumentUri

One LINQ change in CleanChunks:

private IEnumerable<RagChunk> CleanChunks(IEnumerable<RagChunk> chunks)
{
    return chunks
        .Where(c => c.Score >= MinRelevanceScore)
        .OrderByDescending(c => c.Score)
        .GroupBy(c => c.DocumentUri)
        .SelectMany(g => g.Take(1))   // or Take(2) for multi-section docs
        .Take(MaxChunksForSynthesis);
}

 

After the fix:

Chunks retrieved: 3 (down from 8)
Token count sent to synthesis LLM: ~12,266 (down from ~19,700)
Latency: reduced significantly

 

GroupBy(c => c.DocumentUri).SelectMany(g => g.Take(1)) ensures you get at most one chunk per source document before applying the overall Take(MaxChunksForSynthesis) cap.

If documents have meaningful multi-section structure and a single chunk feels too restrictive, Take(2) per group is a reasonable compromise.

The key is that you’re no longer sending five paragraphs from the same page when you have a limit of eight chunks total.

~

What Else the Logging Revealed

Two more things surfaced once instrumentation was in place.

Three LLM calls per request

There were three separate synthesis calls happening for each query, not one. One of those calls was receiving approximately 62,860 tokens, suggesting a code path that bypassed CleanChunksentirely.

This hadn’t been visible without per-call logging. Counting your LLM calls per request matters; in a pipeline with multiple steps, it’s easy to accumulate calls that multiply both cost and latency.

A semantic cache that wasn’t hitting

The system included a semantic similarity cache using KNN search against previously logged queries at a 0.92 threshold, the intent being to skip the vector store and LLM entirely for queries similar to recent ones.

Two problems:

• Cache hits were logged at Debug level. At the default Information log level, you’d never see them. There was no way to know if the cache was working without changing the log configuration.
• The KNN query approach was inconsistent. The cache’s FindSimilarAsync method used .Knn() nested inside .Query(), which applies a different score normalisation than the top-level knn raw JSON approach used in the main vector search. The result: the 0.92 threshold was almost certainly never being met, making the cache effectively dead code.

 

The fixes here are: raise cache hits to Information level from day one, and align the KNN query construction across both paths so scores are comparable.

~

What to Watch Out For

• KNN vector search returns chunks, not documents. If your corpus has paragraph-level chunking, a single document will produce multiple high-scoring chunks with identical or near-identical scores. Without deduplication, your top-N chunks may all come from the same page.
• Token count is the first thing to log in a slow LLM pipeline. Before questioning the model, the infrastructure, or the chunk size, log what you’re actually sending. The answer is usually there.
• Semantic caching is invisible without the right log level. A cache that only logs at Debug is a cache you can’t monitor in production. If you can’t see hits, you can’t tune the threshold, and you can’t tell if it’s doing anything at all.

 

Count your synthesis calls explicitly. One request doesn’t mean one LLM call. Log each call with its token count separately, a single unguarded code path can account for most of your latency.

~

Takeaway

The fix itself ,  GroupBy(c => c.DocumentUri).SelectMany(g => g.Take(1)) , took seconds to write.

The work is in knowing to look for it.  Token count logging before the synthesis call is one of the most useful diagnostic metrics you can add to your RAG pipeline. Audit this from the start.

Once token counts were visible, the bottlenecks stopped being mysterious.

Everything else followed from that.

~