History of Search Engines: From Archie to AI
The history of search engines begins in 1990 with a student project that listed file names, long before most people had heard of the web. This guide follows the story from those first tools through Google and Bing to the AI answers that now sit above search results.
The first search engine was Archie, built in 1990 to index files on FTP servers. Web crawlers and directories followed in the mid 1990s. Google arrived in 1998 with its PageRank idea. Bing launched in 2009, while AI Overviews began appearing in Google results in 2024.
Written by Usman AliGrowth and SEO specialist · 7 years in SEO
Reviewed by Amir ShahidSenior SEO strategist · 9+ years in SEOTimeline
The History of Search Engines on One Timeline
This search engine timeline picks out eight turning points in the history of search engines. Each one changed how people find information, which the sections below explain in more detail.
Archie indexes file names on public FTP servers.
Yahoo launches a web directory sorted by human editors.
AltaVista offers full-text search of the web.
Google launches with PageRank, ranking pages by their links.
Microsoft launches Bing as its new search brand.
RankBrain brings machine learning to Google ranking.
BERT helps Google understand words in the context of a sentence.
AI Overviews add written answers above results.
1990 to 1993
Archie, the First Search Engine
Before the World Wide Web became popular, the internet was mostly a collection of servers that shared files.
In 1990 Alan Emtage, a student at McGill University in Montreal, built a program called Archie with Bill Heelan and Peter Deutsch. It regularly visited public FTP servers, which are computers that let anyone download files. Archie collected the names of every file it found there. Users could search that list to discover where a particular file lived.
Archie could not read what was inside the files, only their names. Even so, it introduced the core idea behind every search engine since, which is to gather information in advance so a search can be answered quickly.
A similar idea soon followed for Gopher, a menu-based system developed at the University of Minnesota. A tool called Veronica, released in 1992, let people search the titles of Gopher menus across many servers. Jughead arrived in 1993 from the University of Utah, offering a similar menu search that usually covered a single Gopher server.
Archie is usually called the first search engine, although it searched file names on the internet rather than pages on the web. The first tools built for the web itself appeared a few years later.
1993 to 1994
1993 to 1994: The First Web Crawlers
Tim Berners-Lee proposed the World Wide Web at CERN in 1989, then opened it to the public in 1991. Within a couple of years there were far too many sites for anyone to track by hand.
W3Catalog
Oscar Nierstrasz at the University of Geneva built it by merging existing hand-made lists of sites into one searchable catalogue.
Wandex
Matthew Gray at MIT wrote the World Wide Web Wanderer to measure the size of the web, then turned what it found into an index called Wandex.
Aliweb
Martijn Koster built Aliweb so site owners could describe their own pages in a special file, which meant the engine needed no crawler.
JumpStation
Jonathon Fletcher launched it at the University of Stirling as one of the first tools to combine a crawler, an index and a search form. Searches only covered page titles and headings, not the full text.
These projects set the pattern later engines followed, which is to visit pages automatically before storing what they find in an index. WebCrawler, written by Brian Pinkerton at the University of Washington, went live in April 1994 as one of the first engines to index the full text of every page. Our web crawler guide shows how modern bots still do this job.
1993 to 1997
Early Web Crawlers and Directories
As the web grew quickly in the mid 1990s, two different approaches emerged. Crawlers used software to visit pages automatically, while directories relied on people to sort websites into categories.
WebCrawler
crawlerCreated at the University of Washington, it was among the first engines to index the full text of web pages instead of just their titles.
Lycos
crawlerDeveloped at Carnegie Mellon University, Lycos grew into one of the best-known search sites of the decade.
Yahoo Directory
directoryTwo Stanford students, Jerry Yang and David Filo, began a hand-picked list of websites sorted into topics.
AltaVista
crawlerBuilt by Digital Equipment Corporation, it offered fast full-text search across a very large index for its time.
Directories gave tidy, trustworthy results, yet editors could not keep up with millions of new pages. Crawlers covered far more ground, although their results were often cluttered because ranking relied heavily on how often a word appeared on a page. Our guide to the types of search engines explains how crawlers still differ from directories today.
1998
Google and PageRank Change the Rules
By the late 1990s, results on many engines were easy to manipulate by repeating keywords. Two Stanford researchers tried a different approach.
Larry Page and Sergey Brin began a research project called BackRub, which studied the links between web pages. Their key idea, later named PageRank, treated a link from one page to another as a kind of vote. Pages linked from many important pages were judged more likely to be useful.
Google was incorporated in September 1998 with that link analysis at its core. Its plain homepage loaded quickly, while its results felt noticeably more relevant than those of many rivals. In 2000 Yahoo began using Google to power its web results, which introduced Google to a much larger audience.
Links are hard to fake at scale
Repeating a word on your own page is easy. Persuading many respected sites to link to you is much harder, which made links a stronger quality signal.
Ranking became a science
PageRank was only the beginning of a much longer story. Google steadily added many more signals, which our page on search engine algorithms explains.
1998 onwards
Paid Search, Microsoft and the Privacy Era
Once search drew millions of visitors a day, companies needed a way to earn money from it without spoiling the results people came for.
| Year | Milestone | Why it mattered |
|---|---|---|
| 1998 | GoTo.com | Bill Gross launched it at Idealab, letting advertisers bid for top spots while paying only when someone clicked, which created pay-per-click advertising |
| 1998 | MSN Search | Microsoft entered search with results supplied by other companies, then switched to its own index in 2005 |
| 2000 | Google AdWords | Text ads appeared beside the organic results, then moved to a pay-per-click auction in 2002 |
| 2006 | Windows Live Search | Replaced MSN Search in September 2006 before being shortened to Live Search the following year |
| 2009 | Bing | Announced in May, then launched in June 2009 as the successor to Live Search |
GoTo.com became Overture in 2001 before Yahoo bought it in 2003, while AdWords grew into the advertising business that still pays for Google Search. Paid listings sit in labelled ad slots on the search results page, so buying ads does not change where a page ranks organically.
Privacy became the next selling point once people grew uneasy about how much engines knew about them. DuckDuckGo arrived in 2008 with a promise not to profile its users, while Startpage passed Google results through without the usual tracking. Ecosia, launched in December 2009, uses the profit from its ads to plant trees around the world.
1997 to 2021
Bing, Regional Engines and Private Search
Google became dominant in many countries, but it never had the field to itself. Other engines grew strong in particular languages or around particular values.
| Year | Engine | Why it mattered |
|---|---|---|
| 1997 | Yandex | Launched in Russia, where it built a strong Russian-language index |
| 1999 | Naver | Founded in South Korea, mixing web results with its own content |
| 2000 | Baidu | Founded in China, where it became the leading engine |
| 2008 | DuckDuckGo | Launched with a focus on not tracking its users |
| 2009 | Bing | Microsoft replaced Live Search with a new brand |
| 2021 | Brave Search | Launched with its own independent index |
Bing became the second major Western index. Under a search deal announced in 2009, Yahoo moved its web results over to Bing, a partnership that still shapes many results pages today. You can see how these engines compare now in our list of the best search engines.
2005 to 2018
Mobile, Maps and Local Search
The arrival of smartphones changed where people searched, along with how often.
Google Maps launches
Online maps made it easy to search for a place. You could see it on a map with directions in seconds. Local results soon became a regular part of search pages.
Universal search
Google began blending images or videos into one results page instead of keeping them in separate tabs. The modern SERP layout grew from this change.
Mobile friendliness counts
Google started using mobile friendliness as a ranking signal for searches on phones. Sites that were hard to use on small screens could lose visibility.
Mobile-first indexing
Google began rolling out mobile-first indexing, which means it mainly looks at the mobile version of a page when deciding how to index it.
Phones also changed the wording of the queries themselves. People started typing or speaking searches such as “coffee near me” while on the move, which pushed engines to treat location as a central part of relevance.
2012 to 2021
The Evolution of Search Engines Toward Understanding Language
Early engines mostly matched the words in a query against the words on a page. Over the 2010s, Google invested heavily in understanding what those words actually meant.
Each step made engines less dependent on exact keyword matches. A search like “can you get medicine for someone pharmacy” became easier to answer once BERT could see how small words connect the rest of the sentence. That shift also changed how people write for the web, since natural, helpful pages now perform better than pages stuffed with repeated phrases.
2023 onwards
AI Overviews and the Next Chapter
The most recent chapter adds generated answers to the results page itself.
In February 2023 Microsoft added a chat assistant to Bing, which could answer questions in full sentences with links to its sources. Google began testing a similar Search Generative Experience the same year. It then launched AI Overviews in the United States in May 2024 before expanding them to many more countries.
These summaries sit above the regular links, so they change how people use the page. Some questions are now answered without a click, while complex questions still send searchers to the sources for detail. AI answer tools such as Perplexity or ChatGPT search have also grown into a new category alongside traditional engines.
For all the change, the foundations from the 1990s remain. Engines still crawl pages before storing them in an index to rank. That is why understanding the core ideas behind search still matters.
FAQ
Questions About Search Engine History
What was the first search engine?
Archie, created in 1990 at McGill University, is generally considered the first search engine. It indexed the names of files stored on public FTP servers.
What was the first web search engine?
Several tools appeared in 1993, while WebCrawler in 1994 was among the first to index the full text of web pages. The exact answer depends on how you define a web search engine.
When was Google created?
Google was incorporated in September 1998 by Larry Page and Sergey Brin, who developed the PageRank idea while studying at Stanford University.
What came before Google?
AltaVista was one of the most popular engines before Google, along with Lycos. The Yahoo Directory was also widely used, with sites chosen by human editors.
How have search engines changed over time?
They moved from matching file names to matching words. Later they learned to analyse links, while today they work to understand language. Recent engines also add AI-written summaries above the results.
Who invented Archie?
Alan Emtage created Archie in 1990 while studying at McGill University in Montreal. He built it with Bill Heelan and Peter Deutsch to index the names of files on public FTP servers.
When did Bing replace MSN Search?
MSN Search was replaced by Windows Live Search in September 2006, a name later shortened to Live Search. Microsoft then announced Bing in May 2009 before launching it in June that year.
Keep learning