Robots.txt Explained: How to Control What Search Engines Crawl
Most site owners never open their robots.txt file. It is a small text file sitting in your root folder, and it quietly decides which parts of your site Google is allowed to look at. Set it up well and your best pages get picked up faster. Set it up badly and you can hide your whole site from search by accident. That happens more often than you would think. This page explains what the file does, how to write one without breaking things, and the errors people keep making. The tool above will build the file for you.
1. What a robots.txt file actually is
It is a plain text file. Nothing more. You put it in the top folder of your site so it loads at yourdomain.com/robots.txt. Any text editor can make one.
Its job is talking to bots. When Googlebot, Bingbot, or any other crawler shows up at your domain, this file is the first thing it asks for. It reads the rules line by line and decides where to go next. If the file is not there, the bot shrugs and crawls everything, which is fine for plenty of small sites. But here is the part people miss: these are requests, not locks. Google and Bing follow them. Scrapers and spam bots often ignore them completely, so never use robots.txt to protect anything.
2. How crawlers read your rules
Search engines do three separate jobs. They crawl, they index, then they rank. Robots.txt only affects the first one. It has no say in whether a page ranks well.
Picture a bot arriving with a stopwatch running. Your robots.txt is the sign at the door telling it which rooms to skip. Skip the pointless ones and it spends its time on pages you actually want people to find. You are not hiding anything here. You are just directing traffic.
3. Crawl budget, and why it matters
Google will not crawl your site endlessly. Every domain gets a rough allowance for how many pages get fetched and how often. People call it your crawl budget. Google works this way so its servers do not hammer yours.
Small sites hardly ever hit the limit. Bigger ones do, constantly. Run a shop with thousands of filter and sort URLs, or a blog with hundreds of tag pages, and bots can burn the whole allowance on pages nobody searches for. Meanwhile the post you published on Monday still is not indexed by Friday. Blocking the dead weight fixes that.
4. The four lines you need to know
The syntax is short. Four directives cover nearly everything you will ever write. Spelling and slashes matter though, so copy them exactly:
- User-agent: names the bot that the rules below apply to. An asterisk (
*) means all of them. - Disallow: tells that bot to stay out of a path. Everything underneath it is blocked too, not just the folder itself.
- Allow: an exception to a Disallow. Use it when you have blocked a folder but need one file inside it to stay open.
- Sitemap: points to your XML sitemap. This one ignores user-agent blocks, so give it its own line at the top or the bottom.
5. Wildcards for finer control
Two symbols let you write smarter rules. The asterisk stands in for any run of characters. Say your internal search results all have a question mark in the URL and you would rather Google skipped them. One line does it: Disallow: /*?*
The dollar sign marks the end of a URL, which is handy for file types. To keep your PDFs out of search, write Disallow: /*.pdf$ and you are done. Google and Bing both understand these. Smaller crawlers sometimes do not, so do not lean on wildcards for anything you really need blocked.
6. The mistakes that cost people traffic
The worst one is a single character. Disallow: / with nothing after the slash blocks your entire site. Every page, gone. It usually happens when someone pushes a staging site live and forgets to change the file. Traffic drops off within a couple of weeks. Read that line twice before you upload.
The second mistake is treating the file as a hiding place. Anyone can read it. Type /robots.txt after any domain and there it is. So listing your admin folder does not hide it, it advertises it. Put private pages behind a password. If you want a page kept out of Google but still reachable by people, use a noindex meta tag instead. Robots.txt and noindex do different jobs, and people mix them up all the time.
7. Using the generator on this page
The tool at the top writes the file for you. Work through it in this order:
- Pick a user-agent. Leave it on All Robots unless you need a rule aimed at one search engine.
- Choose an action. Allow All opens the whole site. Disallow All shuts it. Custom lets you name specific paths.
- List your paths, separated by commas. Things like
/wp-admin/,/cart/,/tmp/. - Paste your sitemap link in full, for example
https://crawlkit.online/sitemap.xml. - Click Create, then copy the output or download it as
robots.txt. Upload that to your root folder, then open Google Search Console and run the robots.txt report. It will flag anything broken before it costs you visitors.
8. Ready-made examples you can copy
Not sure where to start? These three cover most sites. Swap in your own domain and paths.
A normal WordPress site. This is close to what WordPress writes for itself. It keeps bots out of the admin area but leaves open the one file WordPress needs for its own features:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xmlBlocking internal search and a private folder. Search result pages are the classic crawl-budget drain, and there is rarely a reason for Google to index them:
User-agent: *
Disallow: /search/
Disallow: /*?s=
Disallow: /private/
Sitemap: https://example.com/sitemap.xmlA staging or test site. One rule and nothing gets crawled. Just remember to remove it before the site goes live. Section 6 above explains what happens if you forget:
User-agent: *
Disallow: /