What Is Robots.txt? Rules, Examples and How to Test It

Robots.txt is a small text file that tells search engine crawlers which parts of a website they may visit. This guide explains each rule in plain words. It gives you robots.txt examples to copy, then lets you test your own rules in a free tester.

Updated 26 Oct 202611 min readChecked against RFC 9309

Quick answer

A robots.txt file sits at the root of a website, such as example.com/robots.txt. It lists rules for crawlers like Googlebot, saying which folders or pages they should not crawl. It controls crawling, not indexing, so it is not a safe way to hide a page from search results.

Written by Usman AliGrowth and SEO specialist · 7 years in SEO

The basics

What Is a Robots.txt File, in Plain Words

Think of robots.txt as a sign on the front door of your website. It tells visiting bots which rooms are open and which are private.

Every time a polite crawler arrives at a website, it asks for one file first. That file is robots.txt, and it always lives at the top level of the domain. If a crawler wants to visit shop.example.com, it reads shop.example.com/robots.txt before anything else.

The file uses a simple format called the Robots Exclusion Protocol. You name a crawler, then you list paths it should skip. Crawlers from Google, Bing or Apple follow these rules. Bad bots usually ignore them, which is why robots.txt is a guide, not a lock.

Robots.txt is one of the first tools covered in our guide to crawling in SEO. It shapes where a web crawler spends its time on your site.

What it does

Guides crawlers

It tells bots which URLs to skip. That saves server load. It also keeps crawlers away from pages with no search value, like carts or internal search results.

What it does not do

Hide or protect pages

A blocked page can still show up in search if other sites link to it. To keep a page out of results, use a noindex tag instead.

The syntax

The Main Robots.txt Rules Explained

A robots.txt file is made of groups. Each group starts with a User-agent line, followed by the rules for that crawler.

Rule	What it means	Example
User-agent	Names the crawler the group applies to. A star means every crawler.	User-agent: *
Disallow	A path the crawler should not request. An empty value blocks nothing.	Disallow: /cart/
Allow	Opens a path inside a blocked folder. Google and Bing support it.	Allow: /cart/help/
Sitemap	Points crawlers to your XML sitemap. It works outside any group.	Sitemap: https://example.com/sitemap.xml
Crawl-delay	Asks a bot to wait between requests. Google ignores it, though Bing reads it.	Crawl-delay: 10
# comment	A note for humans. Crawlers skip everything after the hash sign.	# staging rules

Two special characters make rules more flexible. A star stands for any run of characters, so /*.pdf matches every PDF link. A dollar sign marks the end of a URL, so /*.pdf$ matches files that end in .pdf only.

When rules clash, Google follows the most specific one. Specific means the rule with the longest path. If an Allow rule and a Disallow rule are the same length, Allow wins. Paths are also case sensitive, so /Shop/ and /shop/ count as two different folders.

Copy and paste

Robots.txt Examples You Can Copy

These are the robots.txt examples most sites need. Change the folder names to match your own website before you use them.

Allow every crawler everywhere

User-agent: * Disallow:

Block the whole site

# Only for staging or test sites User-agent: * Disallow: /

Block one folder, open one page

User-agent: * Disallow: /private/ Allow: /private/welcome.html

Block one crawler only

User-agent: GPTBot Disallow: / User-agent: * Disallow:

A typical WordPress robots.txt

User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://example.com/sitemap_index.xml

Block URL parameters and PDFs

User-agent: * Disallow: /*?sort= Disallow: /*.pdf$

The WordPress example keeps bots out of the admin area. It still allows the admin-ajax file, because some themes need it to load page content. The last line points crawlers to the sitemap, which our XML sitemap guide explains in full.

Try it yourself

Free Robots.txt Tester: Is This Page Blocked?

Edit the rules, pick a crawler, then type any path. The tester applies the same matching logic Google uses.

Robots.txt rules (edit me)

CrawlerPage path to test

Try the path /search?q=shoes, then try /search/help/. The first is blocked by Disallow: /search. The second is open, because Allow: /search/help/ is a longer, more specific match. Switch the crawler to GPTBot, and every path is blocked, since that bot has its own group.

Real-world case

How Robots.txt Became an Official Web Standard

Case study

A 25-year-old rule finally gets written down

Robots.txt began in 1994 as an informal agreement between early web developers. For 25 years, every search engine read the file in its own way. Small differences meant the same file could block a page for one bot while allowing it for another.

1994Martijn Koster proposes the Robots Exclusion Protocol after a crawler overloads his server.
July 2019Google publishes its own robots.txt parser as open source. It also asks for the protocol to become an internet standard.
Sept 2019Google stops reading unofficial rules such as noindex inside robots.txt.
Sept 2022The rules are published as RFC 9309, the formal standard crawlers follow today.

The 2019 change caught many sites off guard. Some had used a noindex line in robots.txt to keep pages out of Google. Once support ended, those pages could appear in search again.

Lesson: only use rules that are part of the standard. To keep a page out of search, put a noindex tag on the page itself.

Avoid these

Common Robots.txt Mistakes That Hurt SEO

Most robots.txt problems come from one wrong line. These are the errors seen most often on real sites.

Blocking the live site

A staging file with Disallow: / gets copied to the live site during launch. Search traffic then slowly drops to zero.

Check after every launch

Blocking CSS or JavaScript

Googlebot needs these files to render your pages. Without them, it may see a broken layout or miss content.

Keep theme files open

Using it to hide pages

Blocked pages can still be indexed from links. Worse, Google cannot see a noindex tag on a page it is not allowed to crawl.

Use noindex instead

Wrong location. The file must sit at the root, like /robots.txt. A file inside a folder is ignored.
Wrong case. Rules are case sensitive, so check every capital letter in your paths.
Treating it as security. Anyone can read your robots.txt. Never list secret folders in it, because you are pointing people right at them.
A file that errors. If robots.txt returns a server error for a long time, Google may slow or pause crawling of the whole site.

Step by step

How to Create and Check Your Robots.txt File

You only need a plain text editor. The whole job takes about ten minutes.

Open a plain text editor and save a file named robots.txt, all in lower case.
Add a User-agent line, then the Disallow or Allow rules you need.
Add a Sitemap line with the full address of your sitemap.
Upload the file to the root folder of your site, so it opens at yourdomain.com/robots.txt.
Open the robots.txt report in Google Search Console to confirm Google fetched it without errors.
Use the URL Inspection tool on an important page to make sure it is not blocked.

Many WordPress SEO plugins let you edit robots.txt from the dashboard, so you may not need to upload anything. Google caches the file for up to 24 hours. A change can take a day to show in its tools.

Once crawlers can reach your pages, the next step is making sure they get stored. Our guide to search engine indexing covers what happens after the crawl.

FAQ

Questions About Robots.txt

Where is the robots.txt file located?

It always sits at the root of a domain, such as example.com/robots.txt. Each subdomain needs its own file, so blog.example.com reads blog.example.com/robots.txt.

Does robots.txt stop a page from being indexed?

No. It only stops crawling. A blocked page can still appear in results if other pages link to it. Use a noindex tag to keep a page out of the index.

Is robots.txt required for SEO?

No. A site without one is treated as fully open to crawlers. Most sites still add one, because it points bots to the sitemap. It also keeps them out of low-value areas.

Does Google follow crawl-delay?

Google ignores the crawl-delay rule. Googlebot adjusts its speed on its own, based on how fast your server responds. Bing does read crawl-delay.

How big can a robots.txt file be?

Google reads the first 500 kibibytes of the file and ignores anything after that. Most real files are only a few lines long, so this limit rarely matters.

Wrapping up

Final Thoughts on Robots.txt

Robots.txt is a short file with a big job. It tells crawlers where they may go. That helps them spend their time on the pages that matter to you. Keep it simple, with a clear group for all bots plus a sitemap line. Only add rules when you have a real reason, like a cart, an admin area or endless filter URLs.

Remember the one rule that trips up most people. Robots.txt controls crawling, not indexing. If you want a page gone from search results, let Google crawl it, then add a noindex tag. Test every change with a tester like the one above. Check the robots.txt report in Search Console after each site launch. One careful look takes minutes. It can save months of lost traffic.

Keep learning

Where to Go Next

/crawling/What is crawling in SEO?How bots find and fetch your pages./crawling/xml-sitemap/XML sitemapPoint crawlers to every important URL./crawling/googlebot/What is Googlebot?How Google’s crawler reads your site./indexing/noindex/Noindex tagThe right way to keep pages out of search./crawling/web-crawler/What is a web crawler?Search bots and AI bots explained./resources/glossary/Search engine glossaryEvery crawling term, defined.