Verified against LlamaIndex · 2026-07-20
Design the ingestion pipeline's metadata schema before you've indexed anything
A prompt for deciding what provenance, access-control, and structural metadata every document needs to carry through parsing and chunking, with a hard rule against silent defaults on missing required fields — so retrieval can filter and cite correctly instead of the whole corpus getting reindexed later to backfill a field nobody planned for.
The prompt
Ready to copy — highlighted parts are example details you can swap.
You are designing the metadata schema that every document and chunk carries through an ingestion pipeline, before ingestion runs at scale — because a metadata field missing at ingestion time means re-processing the entire corpus later to backfill it, while a field that's present from day one costs almost nothing extra to capture. SOURCE SYSTEMS Confluence wiki pages, Google Drive PDFs, and a Zendesk help center, synced via three separate connectors RETRIEVAL-TIME NEEDS cite the exact source and last-updated date in every answer; filter by department-level access before results reach a user; prefer the newest version when a topic has been updated — every metadata field in the schema below needs to trace back to one of these needs; a field with no stated retrieval-time use is either dead weight or a sign a real requirement wasn't thought through yet. METADATA SCHEMA DESIGN For every document, capture: - Provenance: source_system, source_url, original_author, ingestion_connector_version — enough to answer "where did this come from and can I trust it" without leaving the retrieval system, since a chunk with no traceable source is a citation that can't actually be verified. - Access control: department_tags: string[] mirrored from each source system's own sharing permissions at sync time, checked against the querying user's department claims before retrieval, not after — the field(s) that let retrieval filter out documents a given user shouldn't see, applied at query time, not as a post-filter on results already retrieved. A post-filter can leak the existence of a restricted document through result-count side channels even if the content itself is hidden. - Freshness: a captured timestamp for both when the source document was last updated and when it was last ingested — these are different facts, and collapsing them into one "updated_at" field loses the ability to tell which one happened when a staleness complaint comes in. - Structural position: parent_doc_id, heading_path (array of heading strings from root to this chunk), page_number (for PDFs only) — enough hierarchy that a retrieved chunk can be presented with real context about where it sits, not as an orphaned paragraph. HANDLING CHANGES AND DELETIONS - State how a re-ingested document, one whose source was edited, gets handled: are old chunks from the previous version explicitly deleted or superseded, or do stale and current chunks both remain retrievable with no way to tell which is current? Silently accumulating superseded chunks is a specific, common bug that degrades retrieval quality slowly and invisibly as sources get updated over time. - State how a deleted source document is handled — is there an active deletion sweep, or does the corpus just accumulate ghosts that can still surface in retrieval long after the source was removed? VALIDATION a document missing department_tags fails ingestion for that document and is logged to a review queue; it is never ingested with an empty/public-by-default tag — state what happens to a document that's missing a required metadata field at ingestion time: does it fail the batch, get ingested with a null and flagged for review, or get silently ingested with a best-guess default? Only the first two are acceptable; a silent default defeats the entire point of requiring the field. OUTPUT FORMAT 1. The full metadata schema as field name, type, and which retrieval requirement it serves. 2. The re-ingestion/deletion handling rule. 3. The validation/failure rule for missing required fields.
Customize
Optional — swap in your own details for the highlighted parts above.
Why this works
Requiring every metadata field to trace back to a stated retrieval requirement works in both directions that matter for a schema like this. It stops speculative fields that add processing cost and schema complexity for a use that never actually materializes, and — more importantly — it exposes gaps by making a genuine retrieval need with no corresponding field impossible to miss, since the exercise of connecting a stated need to a field forces someone to notice when a stated need has no field actually designed to satisfy it yet. Enforcing access-control filtering at query time rather than as a post-filter on already-retrieved results is a real security distinction, not a style preference. A post-filter that retrieves the top 10 chunks and then removes the ones the user can't see leaks information through the removal itself: if a user's query returns 3 visible results when it would have returned 10 to someone with full access, the missing 7 slots are themselves a signal that restricted content matching that query exists, even though its content was never shown — a form of information disclosure through result-count and timing side channels that a query-time filter, which never retrieves the restricted content in the first place, doesn't create. Splitting 'source last updated' from 'last ingested' into two separate timestamp fields rather than one collapses two genuinely different failure classes into one that's much harder to diagnose. If a user complains that the bot returned outdated information, the fix depends entirely on which of these is true: a source document that was updated recently but hasn't been re-ingested yet, an ingestion pipeline problem, versus a source document that itself hasn't actually been touched in months and the bot is faithfully returning what's genuinely the latest available information, not a bug at all. A single merged timestamp field can't distinguish these two situations, which means every staleness complaint starts from zero instead of from a timestamp that already points at which system to investigate.
Verified against
LlamaIndex 0.13 · 2026-07-20
Unstructured.io 0.18 · 2026-07-20
Changelog
- 2026-07-20 — Initial publish, verified against LlamaIndex 0.13 metadata extractors and Unstructured.io 0.18 connectors.
Building this for real?
This is a free starting point. If you'd rather have AI agents & automation built and running for your business, that's Scult's day job.
EXPLORE AI AGENTS & AUTOMATION
