How search works
How search engines workThe five steps from discovery to rankingComponentsThe parts that make a search engine runCrawlingHow bots find and fetch your pagesIndexingHow a page gets stored so it can be foundCheck if indexedFour ways to confirm a page is on GoogleCrawling and indexing
Web crawlersGooglebot, Bingbot and the new AI botsGooglebotHow Google's crawler reads your siteRobots.txtRules, examples and a free testerXML sitemapCreate one, then submit it to GoogleCanonical tagPick the main version of a pageNoindex tagKeep a page out of search resultsRanking and results
Search engine rankingWhy one page appears above anotherGoogle ranking factorsWhat really matters, grouped simplyPageRankHow links pass value between pagesAlgorithmsTF-IDF, PageRank and modern rankingResults page (SERP)What each block on the page is forAI searchHow AI answers are built from the indexUnderstand
What is a search engineA plain definition with real examplesTypes of search enginesCrawler, meta, vertical, private and AIBrowser vs search engineTwo tools that people often mix upExplore
Best search enginesGoogle, Bing, DuckDuckGo and moreUses of search enginesThe everyday jobs that search handlesHistory of search enginesFrom Archie in 1990 to AI answersSearch engine basics come down to three jobs. A crawler finds pages by following links, an index stores what each page says, then a ranking system orders the best matches for every query. This free guide shows how search engines work with live demos you can try yourself.
Search engines discover pages through links, crawl them with bots, store their words in an index, then rank matching pages for each query.
Search, end to end
Pick a stage or let it play. Each tab shows what a search engine is doing to one real page at that exact moment.
Google cannot visit a page it has never heard of. It learns about new pages mostly through links from pages it already knows, a bit like following road signs to a new shop.
In short: if no link leads to your page, Google may never find it.
A program called Googlebot asks your website for the page, the same way your browser does when you open a link. It only reads what it is allowed to open.
In short: the page has to open properly, or the robot moves on.
Many modern pages are assembled by code after they load. Google opens them in a browser of its own first, so it can read the finished page rather than the raw code.
In short: text that only appears after a click can be missed.
The important words on the page go into a huge index, much like the index at the back of a book. Later, Google can find every page about a topic in a split second.
In short: a page that is not in the index cannot show up in results.
When someone searches, Google compares every page that matches, then orders them by how well each one answers the question.
AI Overviews write a short answer from pages that are already in Google's index, then link to the pages they used as sources.
Feed a starter by mixing equal weights of flour and water, then discard part of it each day. The bakery guide shows the full schedule.
In short: clear, well-organised pages are easier for AI to quote.
Definition
A search engine is a software system that discovers pages on the web, keeps a processed copy of them in an index, then returns a ranked list of those pages for each query.
That definition is the short search engine overview most textbooks on information retrieval agree on. Three abilities have to work together: crawling the web, storing an index of what was found, and ranking results by relevance. Remove any one of them, and you have a search box instead of a search engine.
The difference matters because it tells you where to put your effort. When a product shows results licensed from another company, being indexed by that company is what actually counts.
Google Search is the search engine, while Chrome is the browser Google makes. A browser runs on your device, requests a page you already know, then renders its HTML on screen. A search engine runs on the provider’s servers, where it builds a list of addresses from its index.
You reach one through the other, which is why the two get mixed up so often. Mozilla Firefox, Safari and Microsoft Edge are browsers too, each able to use Google, Bing or another provider as its default search engine.
| Tool | What it does | Examples | Own index |
|---|---|---|---|
| Search engine | Crawls, indexes, then ranks pages for a query | GoogleBingBrave Search | Yes |
| Web browser | Opens a page you already have the address for | ChromeFirefoxSafariEdge | No |
| Web directory | Lists sites that editors picked by hand | Early Yahoo Directory | No |
| AI answer engine | Writes an answer from pages it retrieves | AI OverviewsPerplexity | Usually borrows |
The pipeline
Search runs as a pipeline. Each stage can only use what the stage before it handed over, so a page that fails early never reaches the later steps. This search engine workflow is the most useful idea for fixing any visibility problem.
A bakery publishes its sourdough guide. A link from its homepage is the only way a search engine can learn that the new page exists.
link found · added to the visit listGooglebot follows that link, checks that it is allowed in, then asks for the page. The page loads without an error, so it can move on.
page opened · no errorsGoogle runs the page in a browser of its own, so it sees the same text that a visitor sees on screen.
finished page · text visibleGoogle picks out the important words, then files each one in its index with a note pointing back to this page.
3 words filed in the indexWhen someone searches, Google compares every matching page. A clear answer, backed by links from trusted sites, lifts the guide to the top.
position #4 → #1Before anything can be fetched, the engine needs the address. Most pages are found through a hyperlink on a page that was already crawled, although an XML sitemap, a Google Search Console submission or an IndexNow ping can also add a URL to the queue. A page with no links pointing to it is not penalised; it is simply unknown.
Web crawlers, also called spiders or bots, take URLs from the queue to request them. Googlebot first reads robots.txt to see whether it may fetch the page. Only a 200 response passes content forward, while repeated server errors slow the crawler down across the whole site, which is why these search engine crawling basics matter to site owners.
Many modern pages build their content with JavaScript, so the engine runs that code in a headless browser to see the final text. Rendering costs far more than reading plain HTML. Google queues this work separately, which means text that appears only after scripts run can reach the index later than text in the source.
The rendered text is split into words, a step called tokenization, before those words are written into the index alongside metadata such as language. The engine also picks one canonical URL from each group of duplicates. Because the search engine indexing process is selective, a page can be crawled cleanly yet still be left out.
When someone searches, query processing fixes spelling, expands the words, then works out the intent. The engine pulls matching pages from the index to rank them on relevance, quality, links, freshness or local context. Results arrive in a fraction of a second because the heavy work of crawling and indexing happened long before.
When a page is missing from search, start at discovery and move forward until something breaks. A page blocked from crawling needs a very different fix from a page that was indexed but outranked, so guessing at content changes usually wastes time.
Worked example
Follow a single bakery page through every stage of the pipeline, including the one small mistake that would keep it out of search entirely.
A bakery publishes a page called “sourdough starter guide”. Nothing happens yet, because no search engine knows the address exists.
The homepage links to the new page. Googlebot recrawls the homepage, spots the link, then requests the page and gets a 200 status back.
Google renders the recipe card, then splits the text into tokens such as sourdough, starter, flour and feed. Each token joins its posting list.
Someone searches “how to feed a sourdough starter”. A clear answer near the top, backed by two links from local food blogs, lifts the bakery onto page one.
With a leftover noindex tag, discovery, crawling and rendering still succeed. Indexing stops the page cold, so ranking never sees it.
Components
Every search engine architecture, from Google down to a weekend coding project, uses the same four working parts. Knowing them turns a vague problem into a specific one.
Finds URLs, follows links, then fetches whatever robots.txt allows.
Parses each page, then records which words appear on which pages.
Fixes typos, expands synonyms, then decides what the person wants.
Scores every candidate so the most useful results come first.
// Auto-advancing. Hover or tap a part to hold it.
A fifth layer, the user interface, is the search box plus the results page you see. It sits outside the retrieval machinery, yet it decides how answers are presented. These search engine components map neatly onto failures, since a missing page was either never crawled, dropped by the indexer, skipped by the query processor or outscored by other pages.
Query processing
Ranking cannot start until the engine understands what you typed. In a fraction of a second, query processing turns a few loose words into a structured request that the index can answer.
The query parser splits your text into tokens, spots phrases in quotes, then reads operators such as site: or a minus sign. Spelling is checked at this point too, which is why a typo still finds the right page.
Tokenization breaks text into terms. Stemming maps words like running or runs back to one root, so a page about running still matches. Very common stop words such as the or of carry little weight on their own.
The engine guesses what kind of answer you want. A query like weather tomorrow needs a quick fact, while best laptop for students suggests a comparison. Your location, language or device can shift that guess.
The processed terms are then looked up in the inverted index. The engine collects every page that holds those terms, often thousands of them, before the ranking algorithm puts them in order.
This is how search engines process queries for every search you run. It also explains a familiar puzzle: two queries that look different to you can return nearly the same results, because after parsing they ask the index the same question.
Index lab
A search engine never reads every page when you type a query. Try the trick it uses instead: look words up in an inverted index.
Type words from the three tiny pages on the right. Google never opens each page to check. It reads a word list instead, much like the index at the back of a book, then keeps only the pages that contain all of your words.
That scale explains the design. Google reports that its Search index is well over 100,000,000 gigabytes (Google Search), so reading every page for every query would be impossible. The word the shows the other half of the idea: it appears in every document, so it tells the engine nothing about relevance. That observation led to inverse document frequency, which modern search engine technology still uses.
Ranking
Retrieval narrows billions of pages down to a few thousand candidates. Ranking then puts those candidates in order, using signals that Google groups into five families.
| Signal family | The question it answers | Example signals |
|---|---|---|
| Meaning | What is the person really asking for? | Spelling fixesSynonymsSearch intent |
| Relevance | Does the page cover the topic of the query? | Keyword matchingHeadingsEntities |
| Quality | Can the page be trusted? | BacklinksExpertiseAccuracy |
| Usability | Does the page work for the visitor? | Mobile layoutSpeedHTTPS |
| Context | Who is searching, and from where? | LocationLanguageSettings |
Ranking mixer
Each type of query leans on different signals. Pick a query or drag the sliders to see why the same four pages swap places.
Each type of query leans on different signals. Pick a query or drag the sliders to see why the same four pages swap places.
Illustrative weights for learning only. No search engine publishes its real formula.
Algorithms
People talk about the algorithm as if it were one formula. In practice, search algorithms are layers stacked on top of each other, with the cheapest checks running first.
Classic formulas such as TF-IDF or BM25 measure how well the words on a page match the words in the query.
PageRank, the idea Larry Page and Sergey Brin built Google on in 1998, treats each link as a vote. Modern systems still use descendants of it.
Models such as RankBrain or BERT compare meaning, so a page can match a query that shares none of its exact words.
No outsider knows the precise weighting, which also changes from one query to the next. Freshness matters a great deal for news yet barely at all for a definition, so lists of search engine ranking factors with fixed percentages deserve some doubt. The ranking systems guide on Google Search Central describes real systems instead of guessing at a formula, which makes it a better starting point than any list of ranking factors.
Results
A SERP is no longer ten blue links. Organic search results now share the page with ads, answers and boxes, so ranking first does not always mean being seen first.
These SERP features change how many people scroll down to the blue links, which is why position alone is a weak measure of visibility.
AI search
AI answers sit on top of the same crawl and index, so the foundations in this guide still decide which pages get cited.
AI Overviews, AI Mode, ChatGPT search and Perplexity write answers, yet they still pull those answers from indexed pages. Google describes this as grounding the response in web results. If a page cannot be crawled or indexed, it cannot be quoted as a source either.
Some popular claims deserve caution. Google’s guide to AI features says no special markup or AI text file is required to appear in them. Clear answers near the top of a page still do most of the work, backed by real expertise along with a layout the crawler can read.
The landscape
Search engines differ in how they collect pages and how they present answers. Most fall into one of six types.
Bots crawl the web automatically, building the index without human help.
Google, BingEditors sort websites into categories by hand.
Early Yahoo DirectoryCrawled results mixed with curated listings or extra features.
Most modern enginesSends your query to several engines, then combines the results.
DogpileSearches one kind of content only, such as papers or jobs.
Google ScholarWrites an answer from pages it retrieves, with links to sources.
PerplexityWho owns the index
Hundreds of search engines exist, yet only a handful run their own index. The rest license results from one of them, which decides what you can find.
| Search engine | Index | Known for |
|---|---|---|
| Own | The largest index in most countries | |
| Bing | Own | Also powers Yahoo and much of DuckDuckGo |
| Brave Search | Own | Independent index launched in 2021 |
| Baidu, Yandex, Naver | Own | Market leaders in China, Russia or South Korea |
| DuckDuckGo | Mostly Bing | Search without personal tracking |
| Ecosia | Partner indexes | Funds tree planting from ad revenue |
For most people the five best search engines are Google, Bing, DuckDuckGo, Brave Search and Ecosia. When you pick one, ask whose index sits underneath, because that decides what you can find.
History
Search went from listing file names to writing full answers in about 35 years. These eight moments shaped how it works today.
Archie indexes file names on public FTP servers.
Yahoo launches a hand-built web directory.
AltaVista offers full-text search of the web.
Google launches with PageRank.
Microsoft launches Bing.
RankBrain brings machine learning to Google ranking.
BERT helps Google read whole sentences.
AI Overviews add written answers above results.
Everyday use
Search engines matter because the web has no shared table of contents. They are the main way a page written by a stranger reaches someone who never knew it existed.
what is an inverted indexhow do search engines workgoogle core updatebest laptop for studentslibrary near mebing webmaster toolssite:edu search techniquesSearch skills
Knowing how an engine reads your query helps you find things faster. These operators work in Google and in most other engines.
"how search engines rank"Quotes match the exact phrase only.site:gov crawlingLimits results to one site or domain type.robots.txt -wordpressA minus sign leaves out a word.filetype:pdf indexingFinds a specific kind of file.ranking OR algorithmMatches either word.Judge what you find as well. Check who wrote a page, when it was last updated and which sources it cites before you trust it. University libraries teach the same habits in their research guides, and they work just as well for everyday questions.
For site owners
Most visibility problems come from a single failing stage. Work through these checks in order before you change any content.
SEO basics
Search engine optimization works with the pipeline rather than against it. Each habit below supports one stage, which is why the order you fix things in matters more than any single trick.
Find the words people actually type, then study the results page for each one. Keyword research shows which questions deserve their own page and which ones are already answered well.
Helps retrievalMatch the page type that already ranks. If the top results are step-by-step guides, a sales page will struggle however carefully it is written.
Helps rankingWrite a clear title tag, a meta description that answers the query, one H1 with descriptive subheadings below it. These meta tags shape how your listing looks on the search engine results page.
Helps the SERP listingReturn a 200 status, keep every page linked from somewhere, list it in a sitemap, then make sure robots.txt does not block it. A desktop crawler such as Screaming Frog reveals the same problems a search bot would meet.
Helps crawlingBacklinks from relevant sites still act as votes of trust. The dependable way to earn a backlink is to publish something specific enough that other writers want to cite it.
Helps rankingOrganic traffic shows whether the work paid off. Google Search Console reports clicks, impressions and indexing errors, while Bing Webmaster Tools gives you the same view for Bing.
Covers every stageBetter search visibility rarely needs anything exotic. Organic search results tend to reward pages that are reachable, clearly written and useful for the exact query, so fix the earliest broken stage before you polish the later ones.
Four controls
Each of these tools controls exactly one stage of the pipeline. Mixing them up is one of the most common reasons a page disappears from search, or refuses to leave it.
| Control | What it controls | What it cannot do | Use it when |
|---|---|---|---|
| robots.txt | Whether a crawler may fetch a URL | Remove a page that is already in the index | You want bots kept away from admin areas or endless filter pages |
| XML sitemap | Which URLs you want the engine to discover | Force a page to be crawled or indexed | You publish new pages often or run a large site |
| noindex tag | Whether a fetched page may be stored in the index | Work on a page the crawler is blocked from fetching | A page should stay live for visitors but out of search |
| Canonical tag | Which version of duplicate pages should rank | Bind the engine, since Google treats it as a strong hint | The same content lives at several URLs |
The classic mistake is blocking a page in robots.txt while also adding a noindex tag. The crawler never fetches the page, so it never sees the tag, which means the URL can linger in results for months.
Problem finder
Pick the problem that sounds most like yours. You will see where the pipeline usually breaks, the common causes, plus what to check first.
Try it yourself
Switch conditions off to see exactly where a page would stop, then run the same checks on your real site with these free tools.
Turn off any switch below to see where Google would get stuck. It stops at the first problem it meets, exactly like the real thing.
Run a page through each step and see exactly where it gets stuck.
Make precise searches without memorising any special codes.
Paste your robots.txt file, the rule sheet for search robots, and see which pages it blocks.
Choose a path
Track one is a search engine explained from first principles. Track two turns the same ideas into practical steps for your own website.
For students, teachers or curious minds who want to know what happens after they press Enter.
For site owners and new SEOs who want their pages crawled, indexed, then shown in results.
Myths
Blocking a page in robots.txt removes it from Google.
It only stops crawling. To keep a page out of the index, use a noindex tag on a page the crawler can still fetch.
Submitting a sitemap gets every page indexed.
A sitemap is a hint. Google still decides whether each page is worth storing.
Structured data is a ranking factor.
It can make a page eligible for rich results, which changes how a listing looks rather than where it ranks.
Longer pages always rank higher.
There is no ideal word count. Length only helps when the topic genuinely needs it.
Running paid ads improves organic rankings.
Ads and organic results run on separate systems, so buying one does not lift the other.
Glossary
These are the words you will meet in every search engine guide. Each definition is short enough to remember, yet precise enough to use when you read documentation or talk to a developer.
| Term | What it means |
|---|---|
| Web crawler | A program, also called a spider or bot, that follows links to fetch pages. Googlebot is the best-known example. |
| Crawl budget | How many URLs an engine is willing to fetch from one site within a given period. |
| Index | The engine’s stored copy of processed pages, which it searches instead of the live web. |
| Inverted index | A lookup table that lists, for every term, the pages that contain it. |
| Tokenization | Splitting text into separate terms that the index can store. |
| Stemming | Reducing words to a shared root, so search and searching match each other. |
| Stop words | Very common words, like the or a, that carry little meaning on their own. |
| Retrieval model | The method that decides which indexed pages become candidates for a query. |
| Relevance scoring | Measuring how well a page answers a query, for example with the BM25 formula. |
| PageRank | The link-based scoring method Larry Page and Sergey Brin published in 1998. |
| Backlink | A link from another website to yours, often read as a vote of trust. |
| Canonical URL | The preferred version of a page when duplicates exist. |
| SERP | The search engine results page you see after running a query. |
| Organic search results | Unpaid listings that the ranking algorithm orders, as opposed to ads. |
FAQ
They cover the steps every engine follows: discovering pages, crawling them, storing them in an index, then ranking them for each query. A good search engine 101 lesson also explains the results page, because that is where the work finally shows up.
Bots follow links to find pages. The engine stores the words from those pages in an index. A ranking system then picks the most useful matches when you search.
Crawling fetches a page. Indexing processes the page, then stores it. Ranking compares indexed pages for a specific query to decide the order.
A crawler fetches pages, an indexer stores them, a query processor reads your search, while a ranking algorithm decides the order of the results.
Google Search is a search engine. Google Chrome is the web browser, which is why you can still use Bing or DuckDuckGo inside Chrome.
SERP stands for search engine results page, the screen you see after running a query. Today it mixes organic links with ads, AI Overviews, maps or video results.
Query parsing is the step where the engine breaks your search into terms, reads operators such as quotes, then corrects spelling before it looks anything up in the index.
For most people the shortlist is Google, Bing, DuckDuckGo, Brave Search and Ecosia. Only some of them keep their own index, so results can overlap.
Hundreds exist, yet only a handful run their own large web index. Google, Bing, Brave Search, Baidu, Yandex and Naver are the main ones, while most others license results from them.
A browser such as Chrome opens a page you already have the address for. A search engine such as Google finds that address for you by searching its index.
People use them to look up facts, learn how things work, follow news, compare products, find places nearby, reach known websites or research for school.
Make the page reachable, match the intent shown on the results page, then answer the question fully near the top. Links from relevant sites help once those basics are in place.
Most often the page has not been discovered, is blocked from crawling, or carries a noindex tag. The URL Inspection tool in Google Search Console shows which stage failed. New sites can also take a few weeks before pages appear.
It ranges from a few hours to several weeks. Pages linked from an established site tend to be found sooner. A sitemap entry helps the engine notice the URL, although it does not guarantee indexing.
Google handles most searches in many countries. Bing still matters because it also feeds Yahoo, DuckDuckGo and Microsoft Copilot. In China, Russia or South Korea, local engines such as Baidu, Yandex or Naver lead the market.
Start with how crawling, indexing and ranking work. Then practise on a real site with Google Search Console open beside you. Track two of our free course follows exactly that order.
They change how answers are presented, not how pages are found. AI Overviews draw on pages that are already crawled and indexed, so the same foundations still decide who gets seen.
Recap
If you remember one idea, make it this: search runs as a pipeline. Discovery, crawling, rendering, indexing and ranking each have to succeed in order, so every fix starts with finding the stage that failed. These search engine fundamentals hold for Google, Bing and AI answers alike.
Free PDF
Every diagram from this page plus a one-page cheat sheet, ready for study, class or your team.