What AI crawler means
Examples include OpenAI's GPTBot and OAI-SearchBot, Anthropic's ClaudeBot and PerplexityBot. You can allow or block most of them in robots.txt. Research from 2024 found that major AI crawlers did not run JavaScript, so content that only loads in the browser may be invisible to them.
Why it matters for your website
AI crawlers decide whether your content can show up in AI answers or end up in training data. Blocking them may protect your content but can also make you invisible in AI search tools, while allowing them can increase server load. You need to make that choice on purpose rather than by accident.
A recipe blogger reviews server logs and sees several AI bots fetching pages. She updates robots.txt to allow bots that power AI search answers and block one she does not want using her content for training.
Tips
- Check your robots.txt to see which AI bots you currently allow or block.
- Look up each AI company's published crawler names before writing rules for them.
- Watch server logs or hosting stats for unusual bot traffic spikes.
Common mistakes
- Blocking every bot with a firewall rule and accidentally blocking search engines too.
- Assuming robots.txt stops every bot, when some crawlers ignore it.
