Verified against LlamaIndex · 2026-07-22
Design a chunking strategy before you write a single line of ingestion code
A decision prompt that forces you to pick chunk size, overlap, and splitting method based on the actual document structure and query patterns in front of you, instead of defaulting to a generic 512-token window because it's the number every tutorial uses.
The prompt
Ready to copy — highlighted parts are example details you can swap.
You are a retrieval systems engineer deciding the chunking strategy for a RAG pipeline before any ingestion code gets written. Your job is to pick a concrete chunk size, overlap percentage, and splitting method, and justify each choice against the actual documents and queries below — not to recite the general tradeoffs of chunking strategies in the abstract. DOCUMENT CORPUS 600 internal engineering runbooks and incident postmortems, mostly Markdown with embedded code blocks and a few large tables of error codes REPRESENTATIVE QUERIES 'what's the rollback procedure for the payments service', 'summarize what changed in the Q3 infra migration', 'which error code maps to a rate-limit failure' EMBEDDING MODEL AND CONTEXT BUDGET text-embedding-3-large, 8191 token max input, targeting well under 1000 tokens per chunk for retrieval precision CANDIDATE STRATEGIES TO WEIGH Consider at least these three, don't default to the first one that sounds reasonable: 1. Fixed-size token windows with overlap (e.g. 512 tokens, 15% overlap) — simple, but blind to document structure. 2. Structure-aware splitting (split on headings, list boundaries, table boundaries) — respects the document's own organization, at the cost of variable chunk size. 3. Semantic/recursive splitting (split on paragraph or sentence boundaries, merge upward toward a target size) — balances structure and size, at the cost of extra preprocessing. DECISION RULES - If the corpus contains structured elements (tables, numbered procedures, code blocks), state explicitly whether structure-aware splitting is required to avoid cutting a table or procedure mid-way — a chunk boundary landing inside a table row is a correctness bug, not a minor quality issue. - Chunk size must be justified against the representative queries: if queries ask for a specific fact, smaller chunks with tighter precision are correct; if queries ask for synthesis across a section, chunks need enough size to contain a coherent unit, and retrieval-time context assembly needs to be part of the plan, not an afterthought. - Overlap must be justified numerically, not just "some overlap to be safe" — state the percentage and the specific failure mode it prevents. - Every chunk must carry doc_id, source_url, heading_path, last_updated_at so results can cite back to a real source location, not a floating string with no provenance. - State the actual token count this strategy produces for a representative sample, and confirm it fits inside the embedding model's limits with room to spare — don't pick a number that only works by coincidence with today's model. WHAT TO REJECT Reject any strategy where the justification is "this is the common default" without connecting it to something specific about this corpus or these queries. A chunking strategy that would be identical regardless of what document set it's applied to is not actually a decision. OUTPUT FORMAT 1. Chosen strategy, chunk size, overlap percentage, and splitting method, stated as concrete numbers and rules. 2. One paragraph connecting each number back to the corpus/query evidence above. 3. The metadata schema for each chunk. 4. One documented risk — the failure mode most likely to surface later, and how you'd detect it.
Customize
Optional — swap in your own details for the highlighted parts above.
Why this works
This works because it treats chunk size and overlap as a testable hypothesis tied to the actual query distribution, not a copied default. The 512-token-with-15%-overlap number that shows up in almost every RAG tutorial is a reasonable starting point for exactly one kind of corpus — dense prose answering fact-lookup queries — and actively hurts retrieval on anything else: a corpus of runbooks with embedded tables and numbered procedures will have those procedures silently cut mid-step by a fixed-size splitter that has no concept of a table row or a numbered list item, and a system that ranks those broken chunks past a similarity threshold often returns fluent nonsense assembled from half a table. Forcing the decision to name the corpus's actual structural elements before picking a strategy surfaces that risk before ingestion runs, not after re-ranking eval scores come back flat with no obvious cause. Requiring a numeric overlap percentage tied to a stated failure mode, rather than "some overlap," matters because overlap has a real cost most default configurations never mention: it multiplies embedding and storage volume by roughly the overlap fraction, and it duplicates content across chunks that then compete for the same top-k slots at retrieval time, silently pushing out a genuinely different relevant chunk. A team that can't say what specific fact-split-at-a-boundary failure the overlap number is defending against usually hasn't actually chosen a number — they've copied one. Distinguishing fact-lookup queries from synthesis queries in the decision rules is the single highest-leverage fork in this prompt, because it changes what "correct" retrieval even means: a fact-lookup query wants the smallest chunk that contains the answer with nothing else diluting the embedding, while a synthesis query needs either much larger chunks or a retrieval-time step that reassembles adjacent chunks — and a strategy tuned for one will measurably underperform on the other in a way that's invisible until someone runs an eval against real queries instead of eyeballing a few examples.
What you get back
Chosen: structure-aware splitting on headings and table boundaries, target 400 tokens, 10% overlap, tables kept as single atomic chunks regardless of size (largest table runs ~650 tokens, still under the 8191 limit with margin). Metadata per chunk: doc_id, source_url, heading_path, last_updated_at, chunk_type (prose | table | code). Risk: synthesis queries spanning two adjacent headings may need context assembly across chunks — flagged for a retrieval-time neighbor-stitching step, not solved by chunking alone.
Verified against
LlamaIndex 0.13 · 2026-07-22
Claude Sonnet 4.6 · 2026-07-22
Changelog
- 2026-07-22 — Initial publish, verified against LlamaIndex 0.13 node-parser strategies and Claude Sonnet 4.6.
Building this for real?
This is a free starting point. If you'd rather have AI agents & automation built and running for your business, that's Scult's day job.
EXPLORE AI AGENTS & AUTOMATION
