What Is a Web Crawler? Googlebot, Bingbot and AI Bots Explained
What is a web crawler? It is a program that visits web pages automatically, reads their content, then follows their links to find more pages. This guide explains how web crawlers work, lists the best-known search and AI crawlers, then shows how to control them on your own site.
A web crawler, also called a spider or bot, is software that requests pages from websites, saves what it finds, then follows links to discover new URLs. Search engines use crawlers like Googlebot to build their index, while AI companies use their own bots to gather content.
Written by Usman AliGrowth and SEO specialist · 7 years in SEOThe basics
What Is a Web Crawler, in Plain Words
Think of a crawler as a very fast reader that never stops. It visits a page, notes every link, then moves on to the next address on its list.
The older name, search engine spider, comes from the idea of a spider moving across a web of links. Today most people simply say bot or crawler. Whatever the name, the job is the same: request a page, save a copy, then queue up the new links found inside it.
Search engines depend on crawlers because there is no central list of every page online. The only way to keep an index current is to revisit known pages while discovering new ones. Our guide to crawling in SEO explains that process from the website owner’s side.
Build a search index
Bots like Googlebot or Bingbot gather pages so they can be stored in an index, then ranked when someone searches.
Feed AI models and answers
Newer bots collect pages to train language models, or to fetch fresh sources for AI search answers.
Under the hood
How Do Web Crawlers Work? The Crawl Loop
Every crawler runs the same basic loop thousands of times a second. The details differ between companies, while the pattern stays familiar.
A scheduler decides which URL comes next. It gives priority to pages that change often or that many sites link to, while it spaces out requests to any single server. That politeness matters, because an aggressive bot can slow a small website down for real visitors.
Each request carries a user agent, a short text label naming the bot. Googlebot announces itself as Googlebot, for example, which lets site owners write rules for it. The label is easy to fake, though, which is why verification matters later on this page.
Press Step to send the crawler to the next page in its queue, or press Auto play to watch the whole crawl. It starts at the homepage and follows links one level at a time.
Queue (next up first)
Visited, in order
Search engine bots
List of Web Crawlers Run by Search Engines
These are the web crawler examples you are most likely to see in your server logs. Each one feeds a search engine’s index.
| Crawler | Owner | Main purpose |
|---|---|---|
| Googlebot | Crawls pages for Google Search, mostly with a smartphone version | |
| Bingbot | Microsoft | Crawls pages for the Bing index, which also supplies several partner engines |
| Applebot | Apple | Gathers pages for Siri, Spotlight and Safari suggestions |
| DuckDuckBot | DuckDuckGo | Supports DuckDuckGo’s own search features alongside partner results |
| YandexBot | Yandex | Crawls pages for Yandex Search, popular in Russia |
| Baiduspider | Baidu | Crawls pages for Baidu, the leading engine in China |
Googlebot deserves a special note in this list. Because Google uses mobile-first indexing, most crawling happens with Googlebot Smartphone, which sees your site the way a phone does. Google also runs separate crawlers for images, video and other products, each with its own user agent.
If you want to know how these engines differ in practice, our guide to the best search engines compares them side by side.
The new arrivals
AI Crawlers and Tokens You Should Know
AI companies now run their own bots. Some gather training data for language models, while others fetch pages to cite in AI search answers.
| Name | Owner | What it does |
|---|---|---|
| GPTBot | OpenAI | Collects public pages that may be used to train OpenAI’s models |
| OAI-SearchBot | OpenAI | Finds pages to show as sources in ChatGPT search features |
| PerplexityBot | Perplexity | Indexes pages so they can be surfaced in Perplexity answers |
| ClaudeBot | Anthropic | Collects public pages that may help train Anthropic’s Claude models |
| Google-Extended | A robots.txt token, not a separate bot, that controls Gemini training use |
Google-Extended works differently from every other name in the table. Google’s normal crawlers still fetch your pages, then the token tells Google whether that content may be used for its Gemini models. Blocking it does not remove your site from Google Search or change its ranking.
Splitting training from search lets you make separate choices. A publisher might block GPTBot to stay out of training data while still allowing OAI-SearchBot, so its articles can appear as cited sources. Our guide to AI search covers how those answers are built.
Not all bots are equal
Good Bots vs Bad Bots, How to Tell Them Apart
Plenty of automated traffic is harmless or even useful, yet a share of it is not. The difference usually comes down to honesty and restraint.
Good bots
They identify themselves clearly, follow robots.txt, pace their requests, then publish documentation explaining what they collect.
Googlebot, BingbotGrey bots
They are honest about who they are, yet their purpose may not suit every site, so owners decide case by case.
SEO tools, AI trainersBad bots
They fake their user agent, ignore robots.txt, then hammer servers to copy content or probe for weaknesses.
Scrapers, spam botsRobots.txt only works with bots that choose to respect it. For truly unwanted traffic, site owners turn to firewalls, rate limits or their host’s bot protection, which block requests whatever the bot claims to be.
Common confusion
Crawler vs Scraper, What Is the Difference?
The two words are often used as if they mean the same thing. They overlap in method, although their goals are quite different.
| Web crawler | Web scraper | |
|---|---|---|
| Main goal | Discover pages across many sites | Extract specific data from chosen pages |
| Scope | Broad, following links site to site | Narrow, often a single site or page type |
| Typical output | An index of pages for search | A spreadsheet of prices, names or listings |
| Respect for robots.txt | Expected from reputable crawlers | Varies widely, often ignored |
A crawler asks which pages exist and what they are about. A scraper asks for one exact piece of information, such as every price on a shop’s category page. Many scrapers include a small crawler to find the pages they want, which is why the line can blur.
Take control
How to Control Crawlers With Robots.txt
Robots.txt is the standard way to tell crawlers where they may go. It sits at the root of your domain, so it applies to every page on that host.
The robots.txt example above does three separate things. The first group lets every crawler in, apart from a members folder. The second group blocks GPTBot from the whole site, while the third uses the Google-Extended token to opt out of Gemini training. Google Search keeps crawling normally because Googlebot is not named.
- Match names exactly. Each crawler reads the group with its own user agent name, so check the spelling in the bot’s official documentation.
- Test before you publish. One stray Disallow line can block your entire site from search, which is a mistake that takes weeks to recover from.
- Do not block CSS or scripts. Googlebot needs those files to render pages, so blocking them can make your pages look broken.
Spot the fakes
How to Verify a Real Googlebot With Reverse DNS
Because anyone can copy Googlebot’s user agent, a visit labelled Googlebot is not proof that Google sent it.
The check Google recommends is called a reverse DNS lookup. DNS is the internet’s address book, translating names into numeric IP addresses. A reverse lookup runs the other way, asking which name belongs to a given IP address.
- Take the IP address of the visit from your server logs.
- Run a reverse DNS lookup on that address to get its host name.
- Check that the name ends in googlebot.com, google.com or googleusercontent.com.
- Run a normal forward lookup on that host name to confirm it returns the same IP address.
If both lookups match, the visit really came from Google. Google also publishes lists of the IP ranges its crawlers use, which makes automatic checks easier. Bing offers the same kind of check, with genuine Bingbot addresses resolving to search.msn.com.
New to the bigger picture? The search engine basics guide shows where crawlers fit among the other parts of a search engine.
FAQ
Questions About Web Crawlers
What is an example of a web crawler?
Googlebot is the best-known example, crawling pages for Google Search. Bingbot, Applebot, DuckDuckBot, YandexBot and Baiduspider are other common search crawlers.
Is a web crawler the same as a search engine?
No. The crawler is only one part of a search engine, the part that fetches pages. The index stores them, while the ranking systems decide the order of results.
Are web crawlers legal?
Crawling public pages is a normal part of how the web works. Problems arise when bots ignore site rules, overload servers or copy protected content, which can break terms of service or local laws.
Should I block AI crawlers?
The answer depends on what you want from AI tools. Blocking training bots such as GPTBot keeps content out of future training data, while allowing search bots such as OAI-SearchBot lets your pages appear as cited sources.
How do I see which crawlers visit my site?
Your server access logs list every request with its user agent and IP address. Many hosting dashboards summarise this data, while Google Search Console shows Googlebot activity in its Crawl stats report.
Keep learning