How Search Engines Work Step by Step
A search engine works in five steps. It starts by discovering a URL before crawling the page. After rendering the page, it stores the content in an index so the page can rank when someone searches. This guide walks through how search engines work step by step. You will see a diagram with a real example, along with the mistakes that stop each stage.
Search engines find pages through links or sitemaps before a crawler fetches them. The engine reads the finished page to save its words in an index. For each query, it compares those indexed pages to order the results. A page can only rank after it has passed every earlier step.
Written by Usman AliGrowth and SEO specialist · 7 years in SEO
Reviewed by Amir ShahidSenior SEO strategist · 9+ years in SEOThe process
The Search Engine Process at a Glance
Every search engine, from Google to a small site search, follows the same order. Each step can only work with what the previous one handed over, which is why a problem early in the chain hides a page completely.
Use Next to send one new page through all five steps. Switch on Break this step to see what happens when that step fails.
What the engine does
What can go wrong
| Step | What happens | What usually stops it |
|---|---|---|
| Discovery | Links, sitemaps or redirects reveal a new URL | No page links to it and it is missing from the sitemap |
| Crawling | Googlebot or Bingbot fetches the page | A robots.txt block, a server error or a slow host |
| Rendering | JavaScript runs in a headless browser | Text that only appears after a click or a login |
| Indexing | Words, links and metadata are stored | A noindex tag, duplicate content or thin pages |
| Ranking | Hundreds of signals decide the order | Stronger pages already answer the query better |
Step 1
Step 1: Discovery, How Engines Find New Pages
Before anything else can happen, the engine has to hear about your page. It has no list of every website, so it builds one by following clues.
Most new pages are found through links. Every crawl starts from seed URLs, a list of well-known pages the engine already trusts, before spreading outward link by link. When a crawler revisits a page it already knows, it reads every link on it and adds unfamiliar addresses to a queue. A homepage that links to a fresh article is the fastest way to get that article noticed.
Discovery also happens through a few other routes:
- XML sitemaps. A sitemap is a file that lists the URLs you want found. Search engines read it regularly, although listing a page does not force them to index it.
- Redirects. When an old URL points to a new one, the engine follows the redirect and learns the new address.
- Direct submission. Google Search Console lets you request indexing for a single URL, while Bing supports the IndexNow protocol for instant notification.
- External links. A link from another website can introduce your page before you have told any engine about it.
A page that nothing links to is called an orphan page. Even with a sitemap entry, orphans tend to be crawled less often, so give every important page at least one internal link.
Step 2
Step 2: Crawling, Fetching the Page
Crawling is the moment a bot actually visits. It behaves much like your browser does, only it reads the code instead of looking at the screen.
A crawler such as Googlebot, often called a spider because it moves across the web along links, picks a URL from its queue. Once it confirms the visit is allowed, it requests the page from your server. The server replies with a status code along with the page itself. Only a successful reply lets the page move on to the next step.
robots.txt decides who may enter
Before fetching, the crawler reads your robots.txt file. A Disallow rule for a folder keeps compliant bots out of every page inside it.
Status codes tell the bot what happened
A 200 status code means the page loaded successfully. A 301 sends the bot to a new address, while a 404 means the page is missing. A 5xx error tells the bot the server failed.
Crawl budget limits the visits
Engines avoid overloading a site, so they spread requests out. Large sites with slow servers may find that new pages wait longer for a visit.
Pages are crawled again and again
Popular or frequently updated pages are revisited often. A page that rarely changes may only be fetched every few weeks.
Step 3
Step 3: Rendering, Seeing the Finished Page
Many modern websites build their content with JavaScript after the page loads. Rendering is how a search engine sees that final version.
The first copy a crawler receives is often just the raw HTML. If the main text is written into that HTML, the engine can read it straight away. If a script adds the text later, Google places the page in a rendering queue. It opens the page in a headless version of Chrome to read what appears.
Rendering costs far more computing power than reading plain HTML, so it can happen hours or days after the crawl. Content that only appears after someone clicks may never be rendered at all. The engine is then left with a page that looks almost empty.
Here is a quick test you can run. Turn off JavaScript in your browser before you reload the page. Whatever text is still visible is what every engine can read on the first pass.
Step 4
Step 4: Indexing, Storing the Page
Indexing turns a fetched page into searchable data. It is also the step where the engine decides whether the page deserves a place at all.
During indexing the engine extracts the text to split it into words. It records where each word appears on the page. The title and headings are noted as well, since they show what the page is about. These words go into an inverted index, a huge lookup table that lists every page containing each term. That structure is what lets an engine answer a query in a fraction of a second. The index itself is far too large for one computer, so it is spread across many servers in data centers around the world.
The engine also groups duplicate or near-identical pages together. It picks one version as the canonical URL to show in results. Not every page that gets crawled is kept in the index. Google reports pages it chose to leave out in the Page indexing report inside Search Console.
| Common reason a page is not indexed | What it means |
|---|---|
| Excluded by noindex tag | The page asked to stay out of the index |
| Crawled, currently not indexed | Google fetched it but judged it not useful enough yet |
| Duplicate, Google chose a different canonical | Another URL with the same content was picked instead |
| Discovered, currently not crawled | The URL is known but has not been fetched yet |
Step 5
Step 5: Ranking, Ordering the Results
Ranking happens live, every time someone searches. The engine compares every indexed page that matches the query. It decides which order serves the searcher best.
First, retrieval pulls thousands of candidate pages from the index that contain the query terms or closely related ideas. Ranking systems then score each of those candidate pages. Google groups its main signals into a few families. The meaning of the query comes first, followed by how relevant the content is. Quality matters too, as does the searcher’s context, such as their location or language, which is where personalization shapes the final order.
The highest-scoring pages appear on the search engine results page. They often sit next to extra features, such as a featured snippet or an AI Overview. Our guide to search engine algorithms explains how those scoring layers work in more detail.
No public list gives exact weights for these signals, because the weights change with the query. A recipe search rewards different things than a news search.
Ranking factors
What Ranking Factors Matter Most
Ranking systems weigh hundreds of signals, yet a handful explain most of the order you see. These are the factors Google describes in its own public documentation.
| Factor | What it measures | Why it matters |
|---|---|---|
| Keywords and relevance | Whether the words and topics on the page match the query | A page has to be about the query before any other signal counts |
| Backlinks and PageRank | Links from other sites, weighted by the authority of each linking page | PageRank treated each link as a vote, an idea that later link signals still build on |
| Content quality | Expertise, accuracy and original value | Google asks for content written for people first, not for search engines |
| Page experience | Core Web Vitals, mobile usability and HTTPS | It mostly separates pages that are otherwise equally useful |
| Search intent | Whether the page type fits what the searcher wants | A how-to query favours guides, while a product query favours shopping pages |
| BERT and machine learning | The meaning of the query beyond its exact words | Models such as BERT read words in context, so a page can match without the exact phrase |
Freshness also counts for news or fast-changing topics, while it barely matters for a stable definition. Our guide to Google ranking factors covers each signal in more depth, along with the parts a site owner can actually control.
Results and revenue
How Results Are Displayed, and How Search Engines Make Money
The ranked list is only part of what you see after a search. A modern results page blends several kinds of listings, each one earned or paid for in a different way.
Organic results
These unpaid listings are ordered by the ranking systems described above. Nobody can buy a place among them, so every position has to be earned.
Sponsored ads
Advertisers bid in an auction to appear for chosen keywords, paying when someone clicks. Google marks these listings with a Sponsored label to keep them visibly separate from organic results.
Featured snippets and AI Overviews
A featured snippet lifts a short answer out of one organic result to show it near the top. AI Overviews write a summary instead, linking to the pages they used as sources.
Knowledge panels
These boxes summarise facts about well-known people, places or organisations. The details come from the Knowledge Graph, a database of entities, rather than from a single page.
Local results
Searches with local intent trigger a map with a short list of businesses, often called the local pack. Google says relevance, distance and prominence decide which businesses appear there.
Why search is free to use
Advertising pays for almost all of it. Search ads are Google’s largest source of revenue, which is why the engine can offer search itself at no cost to users.
Buying ads does not improve organic rankings, because the two systems run separately. Our SERP guide shows every feature on a results page with real examples.
Inside one search
How Google Search Works Step by Step, in Half a Second
Most of the work happens long before you type. The part you trigger with a search is fast because the index has already been built.
- As you type a query, Google corrects spelling while predicting what you mean.
- The query is split into terms, with synonyms or related phrases added where they help.
- Query understanding comes next, as Google works out the intent behind your words. You might want a quick fact, or you might want to compare products.
- Matching pages are pulled from the index, usually thousands of them.
- Ranking systems score each candidate, mainly on relevance with quality close behind.
- The results page is assembled from organic results with any ads or helpful features before it is shown to you.
Bing follows the same overall pattern as Google. The differences between engines come from the size of each index. Their ranking systems also weigh signals in different ways, which is why the same query can return different results.
Troubleshooting
Where the Process Breaks, and What to Check First
When a page is missing from search, find the earliest step that failed. Fixing a later step will not help if an earlier one is still broken.
| Symptom | Likely step | First check |
|---|---|---|
| The URL is unknown to Google | Discovery | Add an internal link and include the page in your sitemap |
| Search Console shows a crawl error | Crawling | Test the URL against robots.txt and confirm a 200 status |
| The cached page looks empty | Rendering | Make sure the main text is in the initial HTML |
| The page is crawled but not indexed | Indexing | Remove any noindex tag, then make the content clearly unique |
| The page is indexed but nobody finds it | Ranking | Match the search intent and answer the question near the top |
For a full walkthrough of every concept on this page, start with our search engine basics guide, which also includes live demos of the index and ranking.
FAQ
Questions About How Search Engines Work
How does a search engine work in simple words?
It finds pages by following links before saving a copy of each one in an index. Whenever someone searches, it picks the most useful pages from that index.
What are the steps of a search engine?
The whole process has five steps in total. Discovery comes first, followed by crawling with rendering. Indexing is the fourth step, while ranking is the last. Some guides merge rendering into crawling, which is why you will also see three or four steps described.
How long does it take for a new page to appear in Google?
It can take a few hours or several weeks. Pages linked from an established site are usually found sooner, while a sitemap entry helps the engine notice the URL.
Is crawling the same as indexing?
No, crawling only means fetching the page from a server. Indexing means processing it before it is stored. A page can be crawled without ever being indexed.
Do all search engines work the same way?
They all follow the same overall steps. Even so, each engine runs its own crawler with its own index. That is why Google can show different results from Bing for the same query.
How do search engines rank results?
They score every indexed page that matches the query on signals such as relevance, content quality and backlinks. Machine learning systems like BERT help them judge meaning, while context such as location fine-tunes the final order.
How do search engines make money?
Search engines earn most of their money from advertising shown beside the organic results. Businesses bid on keywords in an auction, paying each time someone clicks a sponsored ad.
Keep learning