Verified against ChatGPT · 2026-07-22
Audit whether your robots.txt and llms.txt actually let AI crawlers in
A line-by-line robots.txt and llms.txt audit that applies the real AI-crawler precedence rules — most-specific User-agent group wins, longest matching path wins — so you get an exact verdict per bot instead of a guess.
The prompt
Ready to copy — highlighted parts are example details you can swap.
You are an AI-crawler access auditor. You understand that robots.txt directives are resolved by two rules, not by reading top to bottom: (1) the most specific matching User-agent group wins over `User-agent: *`, and (2) within a group, the longest matching Disallow/Allow path wins. Apply those two rules literally — do not guess or summarize casually. DOMAIN: https://example.com ROBOTS.TXT: User-agent: * Disallow: /private/ User-agent: GPTBot Disallow: / LLMS.TXT: none found TASK For each of the following AI crawlers, determine whether it can currently fetch https://example.com's homepage, and cite the exact directive (group + line) that decides it: 1. GPTBot (OpenAI — model training) 2. OAI-SearchBot (OpenAI — ChatGPT search and citations, a separate bot from GPTBot) 3. PerplexityBot (Perplexity — live answers) 4. ClaudeBot (Anthropic — training and Claude's web search) 5. Google-Extended (Google — Gemini and AI Overviews grounding, separate from classic Googlebot) 6. CCBot (Common Crawl — feeds many downstream AI datasets) OUTPUT FORMAT A table: Bot | Purpose (training / live search / both) | Allowed or Blocked | Deciding rule (quote the exact line) | One-line fix if blocked. Then answer: does an /llms.txt exist, and if not, what three pages should it list first for https://example.com?
Customize the highlighted detailsoptional — the prompt above already works
Why this works
Robots.txt precedence is famously counter-intuitive: most site owners assume a blanket `Disallow: /` under `User-agent: *` blocks everything, when in fact a bot with its own explicit group is entirely unaffected by it, and ties within a group are broken by whichever path is longest, not whichever line comes first. This is the exact algorithm the site's own AI Visibility Checker runs server-side against a homepage fetch. Forcing the model to quote the deciding line, rather than assert a verdict, turns a guess into a checkable claim — and separating GPTBot (training) from OAI-SearchBot (live ChatGPT citations) surfaces the single most common self-inflicted AI-invisibility mistake: blocking the training bot on purpose while never noticing the search bot got blocked too.
What you get back
| Bot | Purpose | Status | Deciding rule | Fix | |---|---|---|---|---| | GPTBot | Training | Blocked | `User-agent: GPTBot` / `Disallow: /` | Remove or narrow the GPTBot-specific Disallow if training inclusion is wanted | | OAI-SearchBot | ChatGPT search citations | Allowed | No GPTBot-specific rule applies to it; falls through to `User-agent: *` / no Disallow | None needed | | PerplexityBot | Live answers | Blocked | `User-agent: PerplexityBot` / `Disallow: /` | Add an explicit Allow for the paths you want cited | No /llms.txt found. Suggested first three entries: /pricing, /docs, /blog.
Verified against
ChatGPT GPT-5.1 · 2026-07-22
Claude Claude Sonnet 4.5 · 2026-07-22
Changelog
- 2026-07-22 — Initial publish. Added the explicit "quote the deciding line" instruction after testing showed models will otherwise state a verdict without actually checking group-specificity or path-length precedence.
Building this for real?
This is a free starting point. If you'd rather have SEO built and running for your business, that's Scult's day job.
EXPLORE SEO
