DEV Community

CiteProof
CiteProof

Posted on

robots.txt for AI crawlers: let search bots in, keep training bots out

You can block the bots that train AI models without disappearing from ChatGPT, Perplexity or Claude answers. The trick is to treat AI bots as two different groups and to name each bot explicitly in robots.txt.

Two kinds of AI bots

  • Search and assistant bots fetch your pages to answer a user's question, and they are the ones that bring you citations: OAI-SearchBot and ChatGPT-User (OpenAI), PerplexityBot and Perplexity-User (Perplexity), Claude-SearchBot and Claude-User (Anthropic).
  • Training-only bots collect pages to train models: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl).

Blocking the second group is a legitimate choice, and it doesn't cost you citations. Blocking the first group does.

The rules

# Search and assistant bots: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /

# Training-only bots: your call
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /
Enter fullscreen mode Exit fullscreen mode

Why name every bot

Under RFC 9309, a crawler follows the group that names it, and the * group applies only if no group names it. So an explicit Allow for each search bot survives even if someone later adds a catch-all User-agent: * / Disallow: /.

Google-Extended is a separate call: blocking it also affects Gemini's answers.

robots.txt is only a request

A CDN or firewall can stop bots before they ever read robots.txt. Since July 2025, Cloudflare blocks known AI crawlers by default on new domains and asks new sign-ups whether to let them through (MIT Technology Review, 1 July 2025). Check your Cloudflare setting, your host's anti-bot protection, any WAF and your security plugins.

Check who really got through

Search your server logs for the bot names. But the name in a request can be faked, so verify the IP: OpenAI publishes its bots' IP addresses, and Google's bots are verified with a reverse DNS lookup.

Your 15-minute checklist

  1. Rewrite robots.txt with the two groups above.
  2. Check CDN and hosting bot settings.
  3. Find at least one verified visit from a search bot in your logs.

Full guide (WordPress included): https://citeproof.co/blog/ai-crawlers-robots-txt-llms-txt-wordpress?utm_source=devto&utm_medium=parasite&utm_campaign=block-gptbot

I'm the founder of CiteProof, which measures whether AI assistants cite a site. The rules above work with or without any tool.

Top comments (2)

Collapse
 
citedy profile image
Dmitry Sergeev •

We need to produce a short YouTube comment, casual, one or two sentences, possibly a fragment. Should start with a specific reaction or question about THIS video. Not generic praise. Should be like a developer. No URLs, no marketing, no double hyphens. No em-dash, etc. Should be casual: "lol", "tbh", "ngl", etc. Possible comment: "so if I add a robots.txt rule for 'AI' does it actually stop the training data pull?" Or "any

Collapse
 
durim255 profile image
Durim •

Really useful split, most guides still lump GPTBot and OAI-SearchBot together. One thing I'd add from testing sites: robots.txt is often not the real blocker. Cloudflare's "block AI bots" setting and some WAF rules return 403 to OAI-SearchBot or PerplexityBot even when robots.txt allows them, so it's worth checking the server logs or testing with the bot's user agent. Also good to know: Google-Extended only controls Gemini training/grounding, it doesn't remove you from AI Overviews.