What robots.txt means
robots.txt keeps crawlers away from areas like admin pages and search results. It is a request, not a security control, and it does not reliably keep a page out of search results. One wrong line can block an entire site, which is a surprisingly common launch mistake.
Why it matters for your website
Robots.txt guides search engines away from areas that waste crawl time, like admin pages or endless filter URLs. One wrong line, though, can block your whole site from being crawled. It's also public, so it should never be used to hide sensitive pages.
Example
An online store blocks crawling of its internal search result pages in robots.txt, so search engines focus on product and category pages.
Tips
- Check your robots.txt after every launch or migration.
- Use noindex, not robots.txt, to keep a page out of search results.
- Link to your XML sitemap from your robots.txt file.
Common mistakes
- Leaving a staging rule like 'Disallow: /' in place after launch.
- Blocking CSS and JavaScript files that Google needs to render your pages.
Questions about robots.txt
What is robots.txt?
A text file at a site's root that tells search engine and other bots which areas they may or may not crawl.
Does robots.txt keep a page out of Google?
No. Google says a blocked page can still be indexed without its content if other pages link to it. Use noindex or password protection to keep it out of results.
Can I use robots.txt to block AI crawlers?
You can disallow crawlers by their user agent names, and many AI companies publish names you can block. Compliance is voluntary, so not every bot will follow it.
