What Crawling means
If a page cannot be crawled, it cannot be ranked. Broken links, blocked resources and orphan pages with no internal links all get in the way. Good internal linking and a clean sitemap make crawling easy.
Why it matters for your website
If search engines cannot crawl a page, they cannot index or rank it, no matter how good it is. Problems like blocked robots.txt rules, broken links or pages only reachable through scripts can keep your best content out of search. Making pages easy to find is a basic part of being visible.
Example
A new landscaping service page is not showing in Google. The owner finds no other page links to it, so she adds it to the services menu and the sitemap, and Google finds it soon after.
Tips
- Link to every important page from at least one other page on your site.
- Submit an XML sitemap in Search Console and Bing Webmaster Tools.
- Check robots.txt after launches to make sure nothing important is blocked.
Common mistakes
- Leaving a 'Disallow: /' rule in robots.txt from the staging site after launch.
- Creating orphan pages that no menu or internal link points to.
Questions about Crawling
What is Crawling?
How search engine bots discover pages by following links and reading sitemaps.
How do I get Google to crawl my website faster?
Submit a sitemap, link new pages from existing ones and use Search Console's URL Inspection tool to request indexing for key pages. Google does not guarantee timing, though.
What is the difference between crawling and indexing?
Crawling is when a bot visits and reads a page. Indexing is when the search engine stores and organizes that page so it can appear in results. A page can be crawled but not indexed.
