What Is Googlebot? How Google's Crawler Works

Googlebot is the web crawler Google uses to find pages for Google Search. This guide explains how Googlebot visits your site, the different versions it uses, then shows how to check its activity or slow it down.

Updated 26 Oct 20268 min readChecked against Google crawler docs

Quick answer

Googlebot is Google's main crawler. It requests pages, renders them like a modern Chrome browser, then sends what it finds to Google's index. Today it crawls almost every site as a smartphone, because Google uses mobile-first indexing.

Written by Usman AliGrowth and SEO specialist · 7 years in SEO

The basics

What Is Googlebot, in Plain Words

Googlebot is a program, not a person. It visits billions of pages, reads them, then reports back to Google.

Every search result you see started with a Googlebot visit. Before a page can rank, Google has to know it exists. Googlebot is the part of Google that goes out to fetch it. It is one example of a web crawler, and the most important one for most website owners.

Googlebot does not decide where a page ranks. Its job ends once it has fetched and rendered the page. After that, Google’s indexing systems store the content, while ranking systems decide its position later.

Googlebot Smartphone

The main crawler today

It sees your site the way a phone does. Google uses this version for almost every site, because of mobile-first indexing.

Googlebot Desktop

A small share of visits

It still crawls some pages as a desktop computer. Its visits are now far fewer than the smartphone version.

Under the hood

How Googlebot Crawls and Renders a Page

A single Googlebot visit happens in clear steps. Each step can fail, which is why some pages never reach the index.

01DiscoverFind the URL through a link or a sitemap.

02Check rulesRead robots.txt to see if the page is allowed.

03FetchDownload the HTML, up to the first 15 MB.

04RenderRun the page in Chromium to load JavaScript.

05Hand offPass the result on for indexing.

New URLs mostly come from links. That is why internal links matter so much. An XML sitemap adds a second route, while robots.txt tells Googlebot which paths to skip.

Rendering is the step many people forget. Googlebot runs an up to date version of the Chromium browser, so it can read pages built with JavaScript. Rendering may happen some time after the first fetch, though. Important text or links in plain HTML get picked up fastest.

The whole family

Googlebot and Google's Other Crawlers

Googlebot is the best known, but Google runs several crawlers. Each one has its own user agent name.

Crawler	What it does
Googlebot Smartphone	Crawls pages for Google Search as a mobile device
Googlebot Desktop	Crawls some pages for Search as a desktop computer
Googlebot Image	Crawls image files for Google Images
Googlebot Video	Crawls video files for video results
Googlebot News	Crawls articles for Google News
Google-InspectionTool	Fetches pages when you use Search Console testing tools
GoogleOther	Crawls for Google products outside Search

A robots.txt rule for Googlebot also applies to Googlebot Image, Video or News. If you need different rules for images, name Googlebot-Image in its own group.

Real-world case

Googlebot's Switch to Mobile First

Case study

An eight-year move from desktop to phone

Googlebot used to crawl the web as a desktop computer. As more people searched on phones, Google changed the version of a page it looks at first.

2016Google announces mobile-first indexing, using the phone version of pages for ranking.
2019Mobile-first becomes the default for all new websites.
2023Most of the web has moved. Google keeps pushing slow sites to finish the switch.
July 2024Google says Googlebot Smartphone now crawls every site that works on phones.

Sites that hid content on mobile felt this change most. If a block of text appeared only on desktop, Googlebot Smartphone never saw it. That content stopped counting for search.

Lesson: make sure your phone version shows the same content, links or structured data as your desktop version.

Take control

How to Check, Verify or Slow Down Googlebot

You cannot control Googlebot fully, yet you can watch it closely. You can also send clear signals when it causes trouble.

Check its visits

Open Settings in Google Search Console, then Crawl stats. It shows requests per day, response times or errors.

Search Console

Verify it is real

Fake bots copy the Googlebot name. A reverse DNS lookup should end in googlebot.com or google.com.

Reverse DNS

Slow it down

If Googlebot overloads your server, return a 503 or 429 status for a short time. It will back off on its own.

Temporary only

Speed up your server. Googlebot crawls more when pages load fast. A slow server makes it visit less.
Fix broken links. Every 404 or redirect chain wastes a visit that could have gone to a real page.
Do not block resources. CSS or JavaScript files must stay open, or Googlebot cannot render the page properly.

The full reverse DNS steps are on our web crawler page.

FAQ

Questions About Googlebot

How often does Googlebot crawl a site?

It depends on the site. Popular pages that change often may be crawled several times a day. Small or rarely updated pages may wait weeks between visits.

Can I ask Googlebot to crawl a page now?

Yes. Use the URL Inspection tool in Google Search Console, then click Request indexing. Google adds the page to its queue, although there is no fixed time for the visit.

Does Googlebot run JavaScript?

Yes. Googlebot renders pages with an up to date version of Chromium. Rendering can happen later than the first crawl, so key content in plain HTML is still safest.

Is Googlebot the same as Google Search?

No. Googlebot only fetches pages. Other Google systems index those pages, then rank them when someone searches.

Does blocking Googlebot remove my site from Google?

Over time, yes. Google cannot read pages it may not crawl, so they lose their snippets. Many drop out of results completely.

Wrapping up

Final Thoughts on Googlebot

Googlebot is the front door to Google Search. If it cannot reach, fetch or render a page, nothing else in SEO matters for that page. The good news is that Googlebot wants to crawl your site. You mostly need to stay out of its way.

Give it clear paths with internal links plus an XML sitemap. Keep your server fast, so each visit counts. Make the phone version of your site complete, since Googlebot Smartphone does nearly all the work today. Check the Crawl stats report now and then, so you spot errors early. Verify any odd visitor that claims to be Googlebot. Follow these habits, and Google will always have a fresh copy of your best pages.

Keep learning

Where to Go Next

/crawling/What is crawling in SEO?The full crawling guide./crawling/web-crawler/What is a web crawler?Googlebot, Bingbot and AI bots./crawling/robots-txt/Robots.txtTell Googlebot where to go./crawling/xml-sitemap/XML sitemapHelp Googlebot find every page./indexing/check-if-indexed/Check if a page is indexedSee what Googlebot stored./how-search-engines-work/How search engines workWhere crawling fits in.