What Is Crawling in SEO? A Beginner's Guide to Web Crawling
What is crawling in SEO? It is the step where a search engine’s bot visits a page to read its code, so the page can later be indexed. This guide explains how search engines crawl websites, what robots.txt and XML sitemaps control, when crawl budget matters, plus how to spot crawl errors.
Crawling is when a bot such as Googlebot requests a page from your server and downloads its content. A page has to be crawled before it can be indexed, so crawling is the first gate every ranking page passes through.
Written by Usman AliGrowth and SEO specialist · 7 years in SEOThe basics
What Is Crawling in SEO? A Plain Definition
Web crawling is the automated process search engines use to visit pages across the internet, one request at a time.
A crawler, sometimes called a spider or a bot, is a program that asks web servers for pages in the same way your browser does. Instead of showing the page on a screen, it saves the code, reads the links inside it, then queues any new addresses for a later visit.
Crawling is only the second stage of a longer journey. Our guide on how search engines work shows the full order: discovery, crawling, rendering, indexing and ranking. Each stage depends on the previous one, so a page that is never crawled can never reach the results page.
Fetching the page
The bot requests the URL, receives a status code from the server, then downloads the HTML along with files such as CSS or images.
Storing the page
The engine processes what was fetched, decides whether the page is worth keeping, then records its words in the index.
People often mix the two up, yet the difference matters when something goes wrong. A crawled page can still be left out of the index, which is a separate problem covered in our guide to indexing.
Step by step
How Search Engines Crawl Websites, One Request at a Time
Every visit from Googlebot or Bingbot follows the same short routine. It happens millions of times a day across the web, although each single crawl takes only a moment.
The crawler then passes the downloaded page on for rendering and indexing. Pages that change often, or that many other pages link to, tend to be revisited sooner, while quiet pages may wait weeks between visits.
Search engines also take care not to overload a site. If your server slows down or returns errors, crawlers back off, which protects your visitors but delays the crawling of new pages.
Finding URLs
How Crawlers Find URLs, From Links to XML Sitemaps
A crawler can only visit an address it already knows about. Search engines build that list from several sources, although links remain the most important one.
- Internal links. Every link from one of your pages to another is a path a crawler can follow. A clear menu with sensible links between related articles helps bots reach every page you care about.
- External links. When another website links to you, crawlers that visit that site can discover your page through it.
- XML sitemaps. A sitemap is a file listing the URLs you want search engines to know about. One sitemap file can hold up to 50,000 URLs, while larger sites split their list across several files.
- Redirects. When an old address points to a new one, the crawler follows the redirect to learn the new URL.
- Direct notification. Google Search Console lets you request a crawl of one URL, while Bing, Yandex and a few other engines support IndexNow, a protocol for announcing changed pages.
A sitemap is a hint rather than an order. Listing a URL helps the engine find it, but it does not guarantee a crawl or a place in the index. Google also says it ignores the priority and changefreq values in sitemaps, although it can use an accurate lastmod date.
Links must be real HTML links with an href attribute. Google does not click buttons or follow links that only exist inside a script, so menus built that way can hide whole sections of a site.
Permissions
Robots.txt Basics: Telling Crawlers Where to Go
Robots.txt is a plain text file at the root of your domain that tells well-behaved crawlers which parts of the site they may visit.
Pick a crawler, then type a path or tap an example to see whether this robots.txt file lets it in. You can edit the file too, and the answer updates as you type.
- Group it follows
- Deciding rule
The file is organised into separate groups of rules. Each group starts with a User-agent line naming the crawler, followed by Allow or Disallow rules for paths on your site. An asterisk means the group applies to every crawler that does not have its own group. The sample file in the tester above shows this in action, since Googlebot follows its own group while every other crawler follows the asterisk group.
Two points catch many beginners out when they first edit the file. First, robots.txt controls crawling, not indexing, so a blocked URL can still appear in results if other pages link to it. Second, a noindex tag only works when the bot can crawl the page, because it has to read the tag to obey it.
Never use robots.txt to hide private information. The file is public, so anyone can read it, while bad bots simply ignore the rules. Use a password or a login for anything that must stay private.
Server replies
Status Codes That Matter to Crawlers
Every time a crawler requests a page, your server answers with a three-digit status code. That number tells the bot what to do next.
| Code | Meaning | What the crawler does |
|---|---|---|
| 200 OK | The page loaded successfully | Passes the content on for indexing |
| 301 or 308 | The page moved permanently | Follows the redirect, then usually indexes the new URL |
| 302 or 307 | The page moved temporarily | Follows the redirect but may keep the old URL in mind |
| 404 or 410 | The page does not exist | Drops the URL from the index over time |
| 429 | Too many requests | Slows its crawl rate for your site |
| 500 to 503 | The server failed or is unavailable | Retries later, crawling less if errors continue |
A planned outage should return a 503 code, which tells crawlers the problem is temporary. Showing an error message on a page that still returns 200 creates a soft 404, where the engine sees a successful reply with no useful content.
Crawl budget
Crawl Budget: When It Matters and When It Does Not
Crawl budget is the number of URLs a search engine is willing and able to crawl on your site in a given period.
Google describes it as a mix of two things. The crawl capacity limit is how much crawling your server can handle without slowing down. Crawl demand is how much Google wants to crawl, based on how popular your pages are with how often they change.
Very large or fast-changing sites
Google’s guidance points to sites with around a million pages that change weekly, or ten thousand pages that change daily. Large shops with endless filter URLs often fall into this group.
Most small and medium sites
A blog or business site with a few hundred pages is usually crawled fully without trouble. If new pages appear within a few days, crawl budget is not your problem.
When budget is tight, the fixes are practical. Remove or block low-value URLs such as endless filter combinations, fix redirect chains, then make sure your server responds quickly. A faster server lets crawlers fetch more pages in the same amount of time.
Modern websites
JavaScript and Crawling, What Bots Can and Cannot See
Many sites build their pages with JavaScript, which adds content after the first HTML arrives. That can make crawling harder than it looks.
Google crawls the raw HTML first, then places pages that need JavaScript in a rendering queue. Later, a headless version of Chrome runs the scripts to see the finished page. Rendering takes extra time, so content that appears only after scripts run can be indexed later than plain HTML.
Other crawlers may be less capable than Googlebot here. Some smaller search engines, along with several AI crawlers, may read only the raw HTML without running scripts at all. If your main text lives only inside JavaScript, those bots may see an almost empty page.
- Put the main text, headings and links in the HTML the server sends first.
- Use normal links with href values rather than click handlers on buttons.
- Avoid hiding important content behind a click, a scroll event or a login.
Diagnose problems
How to Check Crawling, Plus Common Crawl Errors
Google Search Console is free, so it is the best place to see how Googlebot treats your site. Three reports do most of the work.
URL Inspection
Enter any URL to see when Google last crawled it, whether crawling was allowed, plus the status it received. The live test fetches the page again on request.
Crawl stats
Found under Settings, this report shows how many requests Googlebot made, the response codes it received, then how quickly your server replied.
robots.txt report
This report shows the robots.txt files Google found for your site, when it last fetched them, plus any errors in the rules.
Page indexing
Statuses such as Discovered, currently not crawled point to pages Google knows about but has not fetched yet.
| Crawl problem | Typical cause | First fix |
|---|---|---|
| Blocked by robots.txt | A Disallow rule covers the URL | Edit the rule if the page should be crawled |
| Server error (5xx) | Overloaded or misconfigured hosting | Check server logs, then upgrade or fix the host |
| Not found (404) | A deleted page or a broken internal link | Fix the link or redirect to the closest page |
| Redirect error | A loop or a long chain of redirects | Point the old URL straight to the final one |
| Soft 404 | An empty or error page that returns 200 | Return a real 404, or add useful content |
| Discovered, not crawled | Weak internal linking or crawl limits | Link to the page from strong pages on your site |
For a closer look at the bots themselves, including Googlebot, Bingbot and the newer AI crawlers, read our guide on what a web crawler is. If you are new to the topic, the search engine basics overview puts crawling in context with every other stage.
FAQ
Questions About Crawling in SEO
What is the difference between crawling and indexing?
Crawling is the visit, when a bot downloads the page. Indexing comes after, when the search engine processes the page, then decides whether to store it for search results.
How often does Google crawl a website?
Google does not crawl on a fixed schedule. Popular pages that change often may be crawled several times a day, while quiet pages on small sites might be visited every few weeks.
Can I make Google crawl my page faster?
You can request indexing for a single URL with the URL Inspection tool in Search Console. Linking to the new page from your homepage or a popular article also helps it get found sooner.
Does robots.txt stop a page from appearing in Google?
Not always. It stops compliant bots from crawling the page, yet the URL can still be indexed without its content if other pages link to it. A noindex tag on a crawlable page is the reliable option.
Do I need an XML sitemap?
Small sites with good internal links can be crawled without one, although a sitemap still helps. It is most useful for large sites, new sites with few links, plus sites that publish often.
Keep learning