How big should a RAG chunk be?
Size a RAG chunk to the embedding model that will encode it - commonly 256 to 512 tokens for sentence-transformers - because anything past that model's input window is silently discarded rather than embedded.
Chunk size is an embedding decision, not an LLM one
The usual mistake is sizing chunks against the language model's context window - tens of thousands of tokens - when the number that matters belongs to the embedding model, which is far smaller. A chunk is only useful if it is encoded in full. Anything past the encoder's input window contributes nothing to the vector, so an oversized chunk is not a richer passage; it is a passage whose tail was thrown away.
The failure is silent, which is why it survives
Nothing errors when a chunk is too long. The encoder truncates, returns a vector of the right shape, and the pipeline stores it. We hit exactly this building RAG Converter: a shipped tokenizer.json carried a maximum length of 128 baked into it, so every chunk was measured at up to 128 tokens no matter how long it really was. Every test passed. The only symptom was retrieval that felt slightly wrong.
Sensible starting points
For sentence-transformer style models, 256 to 512 tokens with 10 to 15 percent overlap is a reasonable default, and short factual questions do better at the lower end. Larger windows help multi-step questions that need several facts in one passage. RAG Converter uses 256 tokens with 32 of overlap when embedding semantically, and a larger window when the output is destined for a model with a bigger encoder.
Overlap exists to survive bad cut points
A passage split mid-argument loses the sentence that made it meaningful. Overlap repeats a little of the previous chunk so a fact landing on a boundary appears whole in at least one passage. It costs storage roughly in proportion to the overlap fraction, which is why it is a small percentage rather than a half.
Measure chunk counts, not just success
The cheapest check is a count. A 5,000-character document that produces one chunk has not been chunked - it has been wrapped. We shipped exactly that bug once, and the test that missed it asserted the output was not empty, which one enormous chunk satisfies. Assert a count against a known document instead.