User-agent: * Allow: / # Disallow alt design directions (not for indexing) Disallow: /blog2 Disallow: /blog3 # Internal API endpoints Disallow: /api/ # Internal render target, not a public page. `_partials/` holds the header/footer # HTML fragments that sync_partials.py injects into every page. They are not # standalone documents (no wrapper, no meta description or canonical), and # they cannot carry a robots meta tag (it would be injected into all 400+ pages), # so this is their only gate. # # `flyers/` is deliberately NOT disallowed. Its pages, PDFs and PNGs are kept out # of the index by the `X-Robots-Tag: noindex, nofollow` header in _headers, and a # Disallow here would stop Google from ever reading that header, so a flyer URL # linked from elsewhere could still surface as a bare result. Disallow: /_partials/ # Source template for the generated city pages. Deliberately NOT given a # `noindex` meta tag: generate_pages.py copies this file's verbatim into # every city page, so a robots tag here would deindex all of them. robots.txt is # the only safe gate. It still serves its raw @@TITLE@@ / @@META_DESC@@ # placeholders, which is why it needs one. Disallow: /_city_template.html # ───────────────────────────────────────────────────────────────────────── # AI engine crawlers — explicitly welcomed. # # READ THIS BEFORE EDITING: robots.txt is default-ALLOW. Any crawler not # named below falls through to the "User-agent: *" group above and is # ALREADY permitted — including new, undocumented, or non-Western crawlers # (DeepSeek, Qwen, Kimi/Moonshot, GLM/Zhipu and others do not publish stable # user-agent strings). Naming a crawler here CANNOT block one that isn't # named. This list is a declaration of intent, not a gate. # # The protection for crawlers we don't know about is the permissive # "User-agent: *" group — do not tighten it to a blanket Disallow. # # CCBot is the highest-leverage entry: Common Crawl feeds a large number of # models, including many that never crawl this site directly. # # To revoke a specific crawler later, change its Allow: to Disallow: here. # ───────────────────────────────────────────────────────────────────────── # OpenAI User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / # Anthropic User-agent: ClaudeBot Allow: / User-agent: Claude-Web Allow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: anthropic-ai Allow: / # Google (Gemini / AI Overviews training opt-in) User-agent: Google-Extended Allow: / # Perplexity User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / # Apple Intelligence User-agent: Applebot-Extended Allow: / # Meta AI User-agent: Meta-ExternalAgent Allow: / User-agent: Meta-ExternalFetcher Allow: / # Common Crawl — feeds many downstream models, including ones that never # crawl directly. Highest-coverage single entry in this list. User-agent: CCBot Allow: / # Cohere User-agent: cohere-ai Allow: / User-agent: Cohere-Training-Data-Crawler Allow: / # Mistral User-agent: MistralAI-User Allow: / # Amazon User-agent: Amazonbot Allow: / # Allen Institute for AI User-agent: AI2Bot Allow: / # You.com User-agent: YouBot Allow: / # Diffbot User-agent: Diffbot Allow: / # Webz.io User-agent: Omgilibot Allow: / User-agent: Omgili Allow: / # ByteDance (Doubao) User-agent: Bytespider Allow: / # Huawei User-agent: PetalBot Allow: / # Baidu (Ernie) User-agent: Baiduspider Allow: / # Alibaba / UC User-agent: YisouSpider Allow: / # Qihoo 360 User-agent: 360Spider Allow: / Sitemap: https://pickleland.com/sitemap.xml Sitemap: https://pickleland.com/sitemap-index.xml