llms-txt-and-crawler-access

---
name: llms-txt-and-crawler-access
description: Review crawler and AI-agent access files for ecommerce sites, including robots.txt, sitemap, llms.txt, and AI bot rules.
license: MIT
---

# LLMs.txt And Crawler Access

## Workflow

1. Check `robots.txt` for blocked product/category/policy paths and AI bot rules.
2. Check sitemap availability, freshness, and product/category coverage.
3. Check whether `llms.txt` exists and points to useful public resources.
4. Check canonical URLs and redirects that may confuse crawlers.
5. Recommend minimal changes that improve agent discoverability without exposing private areas.

## Delegation

This skill is the ecommerce-scoped access review. For depth, delegate to the ai-visibility pack instead of duplicating its work:

- Full robots/meta-robots/header analysis → `robots-ai-crawler-audit`.
- Sitemap coverage, canonicals, and redirect chains → `sitemap-discovery-audit`.
- Drafting a new `llms.txt` → `llms-txt-generator`.

## Guardrails

- Never recommend opening admin, cart, checkout session, account, or private order paths to crawlers.
- Keep access recommendations scoped to public storefront evidence.