Noindex vs Robots.txt Disallow: Which One You Actually Need
The short version
- Disallow stops a crawler fetching the page. It does not stop the URL appearing in search.
- Noindex stops the page appearing in search, but only if the crawler is allowed to fetch it and read the tag.
- Using both together breaks the noindex. The crawler never sees it.
- Want a page out of Google? Allow the crawl, add noindex, wait.
These two controls look interchangeable. They are not, and mixing them up is behind one of the most common complaints in technical SEO: a page you deliberately blocked is still showing in Google, usually with the words No information is available for this page where the description should be.
The confusion comes from a single assumption that feels obvious and is wrong: that blocking a crawler from a page removes the page from search. It does not. Here is why.
1. Crawling and indexing are two different jobs
A search engine does three things in sequence. It crawls, meaning it fetches the page. It indexes, meaning it decides to store the page and make it eligible for results. Then it ranks, meaning it decides where the page sits among competitors.
Robots.txt only touches the first step. Noindex only touches the second. Once you hold those apart, the rest follows naturally.
2. What Disallow actually does
A Disallow line in robots.txt tells a crawler not to request the URL:
User-agent: *
Disallow: /private-page/Googlebot reads this and does not fetch the page. It never sees the content, the title, or any tag inside the HTML.
But Google can still know the URL exists. If another site links to /private-page/, or you link to it from your own navigation, Google records the address. It now has a URL it believes is a real page, and no way to look inside it. In some cases it will list that URL in results anyway, using the anchor text of the links pointing at it, with no description.
That is the state Search Console reports as Indexed, though blocked by robots.txt. It is not a bug. It is exactly what you asked for: do not read this page. You never said do not list it.
3. What noindex actually does
Noindex is a tag in the page itself, in the <head>:
<meta name="robots" content="noindex, follow">Or an HTTP header, which is the option for non-HTML files like PDFs where there is no <head> to put a tag in:
X-Robots-Tag: noindexEither way the instruction is unambiguous: you may read this page, you may not list it. Google honours that strictly. A page with a noindex tag that Google can fetch will come out of the index, reliably, usually within a few weeks of the next crawl.
The follow part matters more than people think. It tells Google to keep following links on the page and passing value through them. Use noindex, nofollow and you cut the page off entirely, which is rarely what you want on something like a paginated archive that links to real content.
4. The trap: using both at once
This is the single most common mistake, and it is worth stating plainly.
If you disallow a URL in robots.txt and put a noindex tag on it, the noindex will never work. Googlebot obeys the robots.txt rule, never fetches the page, and therefore never reads the tag telling it to drop the page. The page stays in the index indefinitely, because the instruction to remove it is sitting behind a door you locked.
The fix feels backwards and is correct: unblock the page in robots.txt so the crawler can get in and read the noindex. Once the page has dropped out of the index, you can block it again if you want to save crawl budget, though at that point there is usually no reason to.
5. Which one to use, by what you actually want
| What you want | Use this |
|---|---|
| Page out of search results | noindex, crawlable |
| Stop bots wasting time on thousands of filter URLs | robots.txt disallow |
| Keep a staging site entirely out of Google | HTTP password, not robots.txt |
| Genuinely private or sensitive content | Login or password. Neither control is security. |
| Page deleted for good | Return 410 Gone (or 404) |
| Page moved | 301 redirect to the new URL |
| Duplicate of another page you want to keep | Canonical tag, not noindex |
Two of those deserve a note. Robots.txt is not security, ever. The file is public, so listing /admin-backup/ in it tells every passing scraper exactly where to look. And a duplicate page usually wants a canonical tag rather than noindex, because canonical consolidates ranking signals into the page you kept, while noindex simply throws them away.
6. Getting a page that is already indexed back out
Work in this order:
- Check robots.txt first. If the URL is disallowed, remove that rule. Nothing else will work until you do.
- Add the noindex tag to the page head, or the
X-Robots-Tagheader for a non-HTML file. - Confirm Google can see it. Use the URL Inspection tool in Search Console and look at the rendered HTML it fetched. If the tag is not in what Google fetched, it does not count.
- Request indexing on that URL to push a recrawl rather than waiting for the normal cycle.
- Use Removals for anything urgent. The Removals tool hides a URL from results in about a day, but only for roughly six months. It buys you time, it is not the fix. The noindex is the fix.
Then wait. Removal is not instant even when everything is right, because it happens on Google's recrawl schedule. A few days to a few weeks is normal.
7. The robots.txt noindex directive is dead
You will still find blog posts recommending this:
User-agent: *
Noindex: /private-page/Google supported that unofficially for years and stopped honouring it on 1 September 2019, when it standardised its robots.txt parser. It was never part of the specification. Lines like that are ignored today, which means anyone still relying on them has pages in the index they think are hidden. If you inherited a robots.txt from someone else, this is worth grepping for.
8. Crawl budget is the one good reason to disallow
None of this makes robots.txt useless. It is the right tool when the problem is volume rather than visibility.
A shop with faceted navigation can generate hundreds of thousands of URL combinations, all returning near-identical pages. You do not need each of those crawled, evaluated and marked noindex one by one. Blocking the pattern in robots.txt stops the crawl at the door and leaves the allowance for pages that matter:
User-agent: *
Disallow: /*?colour=
Disallow: /*?sort=
Disallow: /search/The trade-off is that any of those URLs already in the index will stay there, because you have just blocked the crawler from reading the noindex. So the order matters: noindex first, let them drop out, then block the pattern. Crawl budget covers when this is worth the effort, which for most sites is never.
9. Checking your work
Three checks, none of which take long:
- Open
yourdomain.com/robots.txtin a browser and read it. People debug for an hour before noticing the file they edited is not the file being served. - Run the URL through URL Inspection in Search Console. It tells you both whether crawling is allowed and whether indexing is allowed, separately. That separation is the whole point of this guide.
- Search
site:yourdomain.com/private-page/in Google. If it appears with no description, you have the classic disallow-without-noindex situation.
If you are writing or editing robots.txt rules, the robots.txt generator on the home page will produce a correctly formatted file you can copy straight in.