Chunking strategies for RAG: the decision that sets your ceiling
Chunk size, overlap and where you cut affect RAG retrieval quality more than the choice of embedding model, and a bad chunker cannot be rescued by a better model downstream.
Why chunking outranks the model
An embedding model can only encode the passage it is given. If that passage is half a sentence, or three unrelated paragraphs, no model produces a useful vector from it. Fixing chunking usually improves retrieval more than upgrading the model, and it costs nothing to run.
Size: aim for one idea
Around 200 to 500 tokens - roughly 800 to 2000 characters - suits most prose. Small enough that a passage is about one thing; large enough to carry its own context. If you are embedding with a BERT-family model, stay well inside its 512-token window or the tail is silently discarded by the tokenizer.
Overlap: cheap insurance against bad luck
Repeat 10 to 15% of each chunk at the start of the next. A fact that straddles a boundary is otherwise in neither chunk's embedding and becomes unfindable. The cost is some duplicated text; the benefit is that boundary placement stops being a lottery.
Cut on boundaries, never on a count
Split on paragraphs first, sentences second, and words only as a last resort. A chunk that begins mid-word contributes a token of pure noise; one that begins mid-sentence describes a fragment whose subject is in the previous chunk. Fixed-width slicing is the most common cause of retrieval that matches but cannot answer.
Measure with real questions
Write twenty questions you actually want answered, run them, and read what comes back. Retrieval quality is easy to measure and almost never measured; most RAG systems are tuned by vibes and then blamed on the model.