How search works
How search engines workThe five steps from discovery to rankingComponentsThe parts that make a search engine runCrawlingHow bots find and fetch your pagesIndexingHow a page gets stored so it can be foundCheck if indexedFour ways to confirm a page is on GoogleCrawling and indexing
Web crawlersGooglebot, Bingbot and the new AI botsGooglebotHow Google's crawler reads your siteRobots.txtRules, examples and a free testerXML sitemapCreate one, then submit it to GoogleCanonical tagPick the main version of a pageNoindex tagKeep a page out of search resultsRanking and results
Search engine rankingWhy one page appears above anotherGoogle ranking factorsWhat really matters, grouped simplyPageRankHow links pass value between pagesAlgorithmsTF-IDF, PageRank and modern rankingResults page (SERP)What each block on the page is forAI searchHow AI answers are built from the indexUnderstand
What is a search engineA plain definition with real examplesTypes of search enginesCrawler, meta, vertical, private and AIBrowser vs search engineTwo tools that people often mix upExplore
Best search enginesGoogle, Bing, DuckDuckGo and moreUses of search enginesThe everyday jobs that search handlesHistory of search enginesFrom Archie in 1990 to AI answersThere are several types of search engines, each finding information in its own way. Some send bots across the whole web, some rely on human editors, while others borrow results or focus on a single subject. This guide explains every main category with real examples you can try.
The main kinds are crawler-based engines, human-edited directories, hybrid engines, meta search engines, vertical engines, privacy-focused engines, AI answer engines and regional engines. Google is crawler-based, Dogpile is a meta search engine, while Google Scholar is a vertical engine.
Overview
Before looking at each category in detail, it helps to see them side by side. The table sums up how each type gathers its results, with a well-known example of each.
| Type | Where results come from | Example | Best for |
|---|---|---|---|
| Crawler-based | Bots that read the web automatically | Google, Bing | Everyday web searches |
| Human-edited directory | Editors who review sites by hand | The former Open Directory Project | Browsing a topic by category |
| Hybrid | Its own crawler mixed with partner data | DuckDuckGo | Broad results with extra features |
| Meta search | Other search engines, combined | Dogpile, SearXNG | Comparing several engines at once |
| Vertical | One kind of content only | Google Scholar, YouTube | Research, video, jobs or shopping |
| Privacy-focused | Partner or own index, with no tracking | Startpage, Mojeek | Searching without a profile |
| AI answer engine | Several pages, summarised by AI | Perplexity | Quick explained answers |
| Regional | An index tuned for one country or language | Baidu, Naver | Local language searches |
Many services fit more than one row. A single engine can crawl the web, add AI summaries and also promise not to track you, so treat these labels as a guide to its main style.
Pick what you want to do, and the tool shows which type of search engine fits that job, with real examples you can use.
Type 1
When people say search engine, this is usually what they mean. A crawler-based engine builds its own picture of the web with automated programs.
A crawler, also called a spider or bot, is software that visits a page, reads its content, then follows its links to find more pages. Everything it collects goes into an index, which is a vast catalogue of words linked to the pages that contain them. When you search, the engine checks that index instead of the live web, so answers arrive in a fraction of a second.
Runs one of the largest crawlers on the web, called Googlebot, alongside its own ranking systems.
Own indexMicrosoft’s engine uses Bingbot to crawl pages. Its index also supplies results to several partner services.
Own indexBuilds an independent index of its own, which makes it one of the few newer engines that does not rely on Google or Bing.
Own indexRunning a crawler at this scale is expensive, which is why only a handful of companies maintain a full web index. Our guide on how search engines work follows a page from the first crawl through to ranking.
Type 2
Before crawlers became reliable, people organised the web by hand. Directories listed websites in categories, much like the sections of a printed phone book.
Yahoo began as a hand-built list of favourite sites put together by two Stanford students. Its directory grew into a famous guide to the early web before closing in 2014.
A volunteer-run directory where editors reviewed each submitted site. It was a well-known source of curated listings until it shut down in 2017.
Every listing was checked by a person, which kept spam out while making each category easy to browse.
Editors could not keep pace as the number of websites exploded, so listings grew outdated while new sites waited for review.
Large general directories are now mostly gone, although curated lists survive in niche areas such as local business guides or trade associations. Our history of search engines explains how crawlers took over from them.
Type 3
A hybrid engine combines more than one way of gathering results. It might run its own crawler while also licensing results from a larger partner, or blend automated rankings with human input.
Draws its links from a range of sources, including Bing, then adds data from its own crawler for features such as instant answers.
Partner plus own crawlerShows results supplied by larger partners, then uses the income from search ads to fund tree-planting projects.
Partner resultsFor years Yahoo showed its hand-picked directory listings beside results from a crawler, which made it a classic hybrid.
Directory plus crawlerEven the biggest engines keep a human element. Google, for example, works with human quality raters who judge sample results, which helps its engineers test changes before they reach everyone.
Type 4
A meta search engine has no index of its own. It sends your query to several other engines at the same time, then merges what comes back into one list.
One of the best-known meta search engines, launched in the mid-1990s. It gathers results from several major engines into a single page.
Free, open-source software that anyone can run on their own server. It queries many engines while hiding the searcher’s details from them.
Flight search tools such as Skyscanner or Kayak work on the same principle, checking many airlines or booking sites at once.
You see a wider mix of sources, although results can be slower while advanced filters are often limited.
Type 5
A vertical engine searches only one kind of content. Narrowing the field lets it offer filters that a general engine cannot match, such as file size, salary range or publication date.
Image engines find pictures that match the words you type. TinEye specialises in reverse image search, which means you upload a picture to find where else it appears online.
The search bar on YouTube only looks through videos, ranking them by relevance, freshness and viewer behaviour.
These index research papers, theses or medical studies. PubMed focuses on life sciences, while Google Scholar covers almost every field.
Product search engines rank items by price, ratings or availability, so the results are listings you can buy.
Job engines gather vacancies from employer sites, letting you filter by location, salary or working pattern.
The search box on a single shop, library or help centre is also a vertical engine, limited to that site’s own pages.
Type 6
Privacy-focused engines promise not to build a personal profile from your searches. Most do not store your history in a way that identifies you, which also means your results are not personalised.
This is the best-known private option for most people. It shows the same results to everyone searching the same thing in the same place.
No search profilePasses your query to Google on your behalf, so you get Google’s results without Google seeing who asked.
Google results, privatelyA British engine with its own independent crawler, built around a strict no-tracking policy.
Own indexPairs its independent index with a privacy promise, which suits people who already use the Brave browser.
Own indexA French engine that markets itself on privacy, with a strong following in parts of Europe.
European optionAlready covered above, it can be self-hosted so your searches never pass through a company’s servers at all.
Self-hostedPrivate engines protect what happens inside the search itself. Your browser, internet provider or the sites you visit can still record activity, so privacy depends on more than one tool.
Type 7
The newest category writes an answer for you instead of only listing links. These tools use large language models, which are AI systems trained on huge amounts of text to produce human-like writing.
Searches the web for each question, then writes a summary with numbered sources you can click to check.
Chat assistants that can look things up on the web, then reply in a conversation where you can ask follow-up questions.
A generated summary that sits at the top of some Google results pages, with links to the pages it drew from.
AI tools sometimes state mistakes with confidence. Checking the cited sources is the safest habit, especially for health, money or legal topics.
Most AI answer engines still depend on a traditional index underneath. They search first, then summarise, which is why the older categories remain important.
Type 8
In a few countries, a local engine competes strongly with Google. These services understand the local language well, while also offering maps, news or shopping built for that market.
The leading search engine in China, designed around Chinese-language content along with local services such as maps.
A major engine in Russia that also offers email, maps, taxis and other everyday services.
A Korean portal whose results lean heavily on its own blogs, question boards and news sections.
A long-running Czech portal that has kept a loyal audience at home for many years.
Yahoo Japan is another example, since it is run separately from the American Yahoo. For a wider tour, start with our guide to what a search engine is or the main search engine basics page.
FAQ
There is no official count that everyone agrees on. Most guides describe between four and eight groups, such as crawler-based, directory, hybrid, meta, vertical, private, AI or regional engines.
Google is mainly a crawler-based engine with its own index. It also runs vertical tools such as Google Images or Google Scholar, along with AI Overviews on some results pages.
YouTube is a video platform with a built-in vertical search engine. Its search bar only looks through videos hosted on YouTube itself.
A regular engine searches its own index. A meta search engine has no index, so it asks several other engines instead, then combines their answers into one list.
Some of them work partly as search engines. Tools like Perplexity search the web before answering, while a chatbot without web access only uses what it learned during training.
Keep learning