How search works
How search engines workThe five steps from discovery to rankingComponentsThe parts that make a search engine runCrawlingHow bots find and fetch your pagesIndexingHow a page gets stored so it can be foundCheck if indexedFour ways to confirm a page is on GoogleCrawling and indexing
Web crawlersGooglebot, Bingbot and the new AI botsGooglebotHow Google's crawler reads your siteRobots.txtRules, examples and a free testerXML sitemapCreate one, then submit it to GoogleCanonical tagPick the main version of a pageNoindex tagKeep a page out of search resultsRanking and results
Search engine rankingWhy one page appears above anotherGoogle ranking factorsWhat really matters, grouped simplyPageRankHow links pass value between pagesAlgorithmsTF-IDF, PageRank and modern rankingResults page (SERP)What each block on the page is forAI searchHow AI answers are built from the indexUnderstand
What is a search engineA plain definition with real examplesTypes of search enginesCrawler, meta, vertical, private and AIBrowser vs search engineTwo tools that people often mix upExplore
Best search enginesGoogle, Bing, DuckDuckGo and moreUses of search enginesThe everyday jobs that search handlesHistory of search enginesFrom Archie in 1990 to AI answersWhat is a search engine? It is a software system that finds information on the web for you, then lists the most useful pages in answer to the words you type. This guide gives a plain definition, the main parts, real examples and the different types you will come across.
A search engine collects web pages ahead of time, stores them in a giant index, then picks the best matches when you type a question. Google, Bing and DuckDuckGo are well-known examples.
Definition
Think of the web as a library with billions of books and no front desk. A search engine plays the role of the librarian who has already read the shelves.
In simple terms, a search engine is a program that answers a query, which is the word or phrase you type into a search box. It does not go looking across the whole internet at the moment you press Enter. Instead, it has already visited huge numbers of pages, saved what they say, then organised that copy so it can be searched in a fraction of a second.
The result is a list of links called a search engine results page, often shortened to SERP. Each link comes with a title, a short description and sometimes extras such as images, maps or a direct answer. The order of that list is decided by ranking systems that try to guess which pages will help you most.
A useful way to remember it: the engine does its heavy work in advance, so the part you see when you search is mostly a very fast lookup.
Inside the engine
Every engine, whatever its size, is built from the same few pieces. Each one hands its work to the next, much like stations on a production line.
Also called a spider or bot
A program that visits web pages, reads their code, then follows the links it finds to reach new pages.
The engine’s memory
A huge database that records which words appear on which pages, so matches can be found almost instantly.
The decision maker
A set of rules that scores matching pages on relevance, quality and usefulness before putting them in order.
The part you see
The search box with the results page, where your query goes in while the ranked list comes back out.
Each of these pieces has its own quirks, from how often a crawler returns to how an index handles duplicate pages. Our guide to the components of a search engine looks at every part in more depth.
Type a search or tap an example to watch a tiny engine look up your words in its index, then rank six sample pages.
Pages that contain more of your words come first. After that, each word scores 3 points in the title plus 1 point for every time it shows up in the text. Real engines weigh hundreds of signals, yet they start with the same lookup.
Common mix-ups
Beginners often use these words as if they mean the same thing. They are related, yet each one does a different job.
A website is a collection of pages owned by one person or organisation. A search engine helps you find websites, although the search engine itself is also a website you visit.
A database holds organised data, such as a shop’s product list. A search engine relies on a special kind of database, its index, which it builds by reading the public web.
A browser is the app on your device that displays web pages, such as Chrome or Safari. You can use a browser to reach a search engine, yet the two are separate tools.
The search bar inside a single shop or blog only looks through that site’s own pages. A web search engine looks across millions of sites at once.
The browser question causes the most confusion, partly because Google makes both. Our page on the search engine vs browser difference explains it with everyday examples.
Real examples
Google is the name most people know, although several other engines serve millions of searchers in different ways.
Launched in 1998, it runs one of the largest web indexes. Its results often include maps, videos, shopping listings or an AI Overview.
General web searchMicrosoft launched Bing in 2009 as the successor to Live Search. It is the default search in the Edge browser, while its results also power several smaller services.
General web searchA privacy-focused engine that does not build a personal profile of you. Its results come from several sources, including Bing.
Private searchOne of the oldest names on the web. Today its standard web results are largely supplied by Bing, with Yahoo adding its own news content.
Web portalThe leading search engine in China, built around Chinese-language content. It also offers maps, an encyclopedia and other services.
Regional engineReleased by the company behind the Brave browser, it uses its own independent index instead of renting results from a larger engine.
Independent indexOther well-known options include Yandex in Russia, Naver in South Korea, Ecosia, which funds tree planting with its ad income, and Perplexity, which writes answers with sources attached. Each one suits a slightly different kind of searcher, so it is worth trying more than one before settling on a favourite.
Categories
Not every engine works the same way. Some crawl the web on their own, while others borrow results or focus on a single kind of content.
| Type | How it works | Example |
|---|---|---|
| Crawler-based | Bots read the web automatically to build an index | Google, Bing |
| Human-edited directory | People review sites before listing them by category | The former Yahoo Directory |
| Meta search engine | Sends your query to other engines, then combines the answers | Dogpile |
| Vertical search | Covers one topic, such as images, jobs or research papers | Google Scholar |
| Privacy-focused | Avoids tracking or storing your search history | DuckDuckGo, Startpage |
| AI answer engine | Writes a summary answer from several sources | Perplexity |
Many engines combine more than one approach, so these labels describe a main style rather than a strict box. The full guide to types of search engines covers each category with more examples.
The short version
Under the surface, a search engine repeats the same routine for every page on the web. Here is the whole cycle in the briefest form.
Each step depends on the one before it, which is why a page that is never crawled can never rank. Our step-by-step guide on how search engines work adds rendering to the picture, with a diagram of every stage.
Why it matters
For many people, a search box is the first stop for almost any question. That habit gives search engines a quiet influence over what we read, buy or believe.
Search engines make the open web usable. Without them, you would need to know the exact address of every page you wanted, which is how much of the early internet felt before automated search arrived.
Because so much attention flows through results pages, a whole field called search engine optimisation has grown around them. For a wider tour of the topic, start from the top with our search engine basics guide.
After you search
The page that appears after a search is more than a plain list of links. Modern engines mix several kinds of results to answer you faster.
These are the ranked links chosen by the engine’s algorithm. No one can pay to appear here, which is why they are called organic.
Advertisers bid to show their links for certain searches. They are usually marked with a small label such as Sponsored.
Boxes that answer directly, show nearby places or highlight a video. They often sit above the regular links.
Some engines now write a short generated answer at the top, with links to the pages it drew from.
Learning to read these elements helps you judge which results to trust. It also explains why two people can see slightly different pages for the same query.
FAQ
It is a tool that finds web pages matching the words you type, then lists them in order of how useful they seem. It works from a stored copy of the web called an index.
Google Search is a search engine that finds web pages for you. Google also makes a browser called Chrome, which is a separate app used to open web pages, including Google Search itself.
Archie, created in 1990 at McGill University, is usually named as the first. It searched lists of files on public servers, since the web as we know it did not exist yet.
Most web search engines are free for searchers because they earn money from ads. A few privacy-focused services offer paid plans with no advertising at all.
Each engine keeps its own index and uses its own ranking rules. Some also borrow results from a larger partner, so overlap is common although never complete.
Keep learning