Real file from a public repo · synced Aug 26, 2026
site-crawlability
When the user wants to improve crawlability, fix orphan pages, or optimize site structure for search engines. Also use when the user mentions "crawlability," "crawl budget," "orphan pages," "internal links," "site structure," "site crawlability," "infinite scroll," "pagination," "masonry SEO," "AI crawler optimization," "GPTBot crawlability," "ClaudeBot crawlability," or "content not indexed." For internal links, use internal-links.
Works with
Install in three steps
- 1
Download the ZIP
One file per agent, plus INSTALL.md with the exact paths below.
- 2
Claude Code: copy SKILL.md into your skills folder
.claude/skills/site-crawlability/SKILL.md # this project ~/.claude/skills/site-crawlability/SKILL.md # every project
- 3
Other agents: use the matching file
skill.mdc goes in Cursor’s .cursor/rules/. AGENTS.md works as GitHub Copilot instructions, a legacy .cursorrules, or a Codex / Gemini CLI instructions file.

