RAG Chunker Fast Path That Measured a Placeholder Instead of 40 KB of Code

In this blog, I run through a debug session that took me a while to diagnose.

Documents in a production RAG pipeline were intermittently failing to vectorise, and the pages that failed were the small ones.

A few hundred characters of prose wrapped around a single big fenced code block.

Every size check in the pipeline should have been split into several chunks.

Instead they arrived at the embedding API as one chunk of tens of thousands of characters, got rejected as too large, and took the whole document’s ingestion down with them.

The cause wasn’t where the symptom was. The embedding API threw the error, but the bug was a length check made early in the chunker, against the wrong thing entirely.

This is a different bug from the missing length-guard-before-the-embedding-call problem I wrote about a couple of weeks back.  This one is about the chunker’s own fast path lying to itself.

If you run a documentation-heavy corpus through RAG, you may have at least one of these three problems.

Let’s dig in.

~

The Setup and a Reasonable Optimisation

A common and sensible chunker trick: pull the fenced code blocks out of the content, replace each with a short placeholder, and reason about the prose without code getting in the way.

Code blocks are dense and noisy, getting them out of the splitting logic makes everything downstream simpler. Nothing wrong with the idea really.

The problem is what you measure after you’ve done the swap.

Let’s look at the bugs.

~

Bug 1: the fast path measured the placeholder, not the content

The chunker had a fast path check

– “is this already small enough that I don’t need to split it?”

Skip the work if the content fits. The trouble is that the check ran after the code blocks had been swapped out for placeholders, and it measured the placeholder version.

So a page with 300 characters of prose and a 40 KB code block looks tiny: prose plus a couple of [[CODE_BLOCK_1]] tokens. It comfortably passes the size check, skips splitting, and then the real code gets swapped back in before the chunk is returned.

One giant chunk, straight to the embedding API.

// Before (buggy): the decision is made against placeholder text
var processed = ExtractCodeBlocks(content, out var blocks);

if (processed.Length <= maxChunkSize)
    return new[] { RestoreCodeBlocks(processed, blocks) }; // giant chunk slips through

// After: decide against what you will actually send
var restoredLength = processed.Length
    + blocks.Sum(b => b.Length - PlaceholderLength);

if (restoredLength <= maxChunkSize)
    return new[] { RestoreCodeBlocks(processed, blocks) };

That’s the whole fix for bug 1:  – compute what the restored length will actually be, placeholder length subtracted, real block length added back, and make the fast-path decision against that.

The code above is illustrative of the pattern rather than verbatim, but the shape is the point: you decided to skip splitting based on a number that doesn’t exist anywhere in the output you produced.

~

Bug 2: “never split a code block” had quietly become “no size limit”

The chunker also had a rule:

– never split a code block.

That instinct is correct.  A code example sliced in half mid-block is useless to a human reader and actively confusing to the embedding model.  You want the whole example, braces balanced, in one chunk.

But “never split” had drifted into “no size cap whatsoever.”

Any block, however far over the embedding model’s token limit, was emitted whole.

So even once the fast-path measurement was fixed, a single 40 KB code block on its own would still blow the limit, because the rule said you weren’t allowed to touch it.

The fix is to keep the guarantee where it’s cheap and drop it only where you have no choice. Cap code blocks. A block over the limit gets split at a safe boundary, a closing brace, a blank line -rather than emitted whole or silently dropped:

// Atomic protection, but with a cap.
// Keep the block whole when it fits; split at a safe boundary when it can't.
if (block.Length <= maxChunkSize)
{
    yield return block;
}
else
{
    foreach (var part in SplitAtSafeBoundary(block, maxChunkSize)) // closing brace / blank line
        yield return part;
}

You only lose the “never split” guarantee for blocks that were too big to keep in one piece anyway. A much preferred option over a failing the document.

~

Bug 3: 4 chars per token is a prose number, and code isn’t prose

The chunker used the standard 4-characters-per-token approximation to estimate token counts. A reasonable assumption,  running a real tokeniser on every chunk adds latency and a dependency, and for prose the ratio holds up well.

It does not hold for code though. Code, markup and XML etc, run closer to 2–3 characters per token because they’re dense with special characters, short identifiers and whitespace.

So a code chunk that looks comfortably within budget by character count can be roughly double that in real tokens, and sail past the model’s actual limit.

You don’t necessarily need a full tokeniser to fix this. Be conservative where it counts:

• Apply a tighter character budget to chunks you’ve identified as code-heavy, assume the worse ratio for them.
• Treat the char-based estimate as a soft limit, and give anything near the boundary more careful handling rather than waving it straight through.

~

What to Watch Out For

Some gotchas to consider or to watch out for:

Decide against what you’ll actually send

Any time you substitute a stand-in for real content to make a decision easier — a placeholder, a summary, a truncated preview, make the decision against the real thing, not the stand-in. This is the bug in one sentence, and it generalises well beyond chunkers.

Atomic protection needs a cap

“Never split X” is fine until X is bigger than the thing it has to fit into. Give every “never split” rule a size ceiling and a safe-boundary fallback.

The 4-chars-per-token rule is a prose number

For documentation-heavy corpora, code, XML, markup etc,  budget tighter, roughly half.

The symptom and the cause can be in different layers

The rejection came from the embedding API; the decision that caused it was made early in the chunker. Trace size related ingestion failures back to where the size decision actually happened, not just where it surfaced.

One bad chunk fails the whole document, silently

Pair chunker fixes with an ingestion-side guard  -a hard size check before the embedding call and not just query-side handling. Belt and braces and all that.

~

Summary

None of these three bugs were complex.   A measurement against a stand-in, an unbounded “never split” rule, and an approximation applied outside its valid range but together resulted in short pages producing enormous chunks.

If you’re building RAG over documentation, go and check what your fast path is actually measuring before it decides not to split.

The answer might not be the content.

~

Enjoy what you’ve read, have questions about this content, or would like to see another topic?

You can schedule a call using my Calendly link to discuss consulting and development services.

~

Courses

Check my AI courses.  From developers to decision makers, these have you covered:

• Developing an Artificial Intelligence Strategy for Your Organization
• Aligning Generative AI with Business Cases
• Vector Databases & Embeddings for Developers

~