A robots.txt file sits at the root of your website (example.com/robots.txt) and tells crawlers which parts of the site they may visit. It's one of the simplest SEO files, and one of the easiest to get disastrously wrong: a single stray Disallow: / can hide a whole site from Google. It's also become the main way site owners express a choice about AI crawlers. OpenAI's GPTBot, Google-Extended (which controls use of your content in Gemini models), Anthropic's ClaudeBot, Common Crawl's CCBot, Perplexity, Apple's Applebot-Extended, Meta and others all publish user-agent names you can allow or block. This free robots.txt generator builds a correct file from simple choices. Set the default for all crawlers, list folders to keep out of search such as /admin/, /cart/ or /search, add your sitemap URL, and choose, one by one or all at once, whether to allow or block each AI crawler. The output follows the Robots Exclusion Protocol (RFC 9309) with a separate group per bot. Two honest caveats: robots.txt is a request, not a lock, so well-behaved bots follow it and bad ones don't, and blocking a page in robots.txt doesn't remove it from Google. Use noindex for that. Generated in your browser.
Generate robots.txt
AI crawlers
How to use it
- Leave the default as Allow unless it’s a staging site.
- Add private or low-value paths to block.
- Choose your AI crawler policy.
- Upload the file as /robots.txt at your domain root, then test it in Google Search Console.
On Blogger
Settings → Crawlers and indexing → Enable custom robots.txt, then paste the result.
FAQ
Does blocking GPTBot remove my content from ChatGPT?
It stops future crawling by that bot. It doesn’t remove anything already collected, and other routes (like search features) may use different user agents.
Does Google-Extended affect my Google rankings?
No. Google says it controls use for Gemini training and grounding, not Search ranking.
Will robots.txt hide a page from Google?
Not reliably. A blocked URL can still be indexed from links. Use a noindex meta tag (and let Google crawl it).
Is the user-agent list complete?
It covers major, documented AI crawlers. New ones appear regularly, so check providers’ docs.
Comments
Post a Comment