Meta Robots Tag Generator
Tick what you need. The tag, the HTTP header and the server config are written for you. Everything updates as you type, in your browser.
Tick a directive above and the tag will appear here.
What the meta robots tag controls
Robots.txt works on paths and whole folders. It decides whether a crawler may fetch a URL. The meta robots tag works on one page and decides something robots.txt cannot touch: whether that page is allowed into the index, and what the search result looks like once it is there.
Those are different jobs, and confusing them is behind most of the trouble people have here. A page can be perfectly crawlable and still kept out of results. A page can be blocked from crawling and still appear in results, listed with no description. Noindex vs disallow covers that distinction properly.
The two places it lives
In HTML, a tag in the <head>. As an HTTP response header, for anything without a head to write into. They are equivalent for a web page; the header is the only option for a PDF, an image or a generated download, which is why this tool emits both along with the server config to apply it.
If both are present and they disagree, the more restrictive instruction wins. Setting both is asking for confusion later.
Why the warnings matter more than the tag
Writing the tag is easy. Writing one that does what you meant is where it goes wrong, and the failures are quiet: no error appears, the tag simply has no effect, or the opposite effect.
The tool flags every combination below as you build it.
index, followdoes nothing. Both are the default. This is the most widely copied piece of cargo-cult SEO there is.nosnippetwithmax-snippet. Contradictory.nosnippetwins and the character limit is ignored.max-snippet:0isnosnippet. Same instruction, written twice.- Snippet settings on a noindexed page. There is no result, so there is nothing to control.
allalongside any restriction. Google takes the restrictive one, soallis doing nothing.indexifembeddedon its own. It only means something next tonoindex.nofollowon a listing page. It cuts off every link on the page, including to the content you want found. Rarely worth it.
The mistake that undoes all of it
None of these directives work on a page Googlebot cannot fetch. If the URL is disallowed in robots.txt, the crawler never loads the HTML, never sees the tag, and never receives the instruction. The page stays in the index, because the request to remove it is sitting behind a locked door.
The fix feels backwards and is right: unblock the page so the crawler can read the noindex. Once it has dropped out, block it again if you want to save crawl budget, though by then there is usually no reason to.
What to put where
| Page | Suggested | Why |
|---|---|---|
| Normal content page | No tag at all | The defaults are already what you want |
| Internal search results | noindex, follow | Infinite URLs, but they link to real pages |
| Thin tag or date archive | noindex, follow | Keeps link value flowing to the posts |
| Filtered or sorted listing | Canonical, not noindex | Canonical keeps the ranking value; noindex discards it |
| Thank-you page | noindex, nofollow | Links nowhere useful |
| Staging site | A password | A tag is not access control |
| Private PDF | X-Robots-Tag: noindex | No head to put a tag in |
| Expired event listing | unavailable_after, then 410 | Drops out on a date you choose |
Note the filtered-listing row. A filtered page is usually a duplicate with an obvious parent, so a canonical tag serves you better than noindex: it consolidates the ranking value into the page you kept, where noindex throws it away.
Checking it afterwards
View source and search for name="robots". Confirm there is exactly one. Two tags with different values is a common WordPress problem, where the theme emits one and an SEO plugin emits another, and Google then has to pick.
For the header version, look at the response directly with curl -I. Then confirm in Search Console with URL Inspection, which reports indexing allowed separately from crawling allowed — exactly the distinction this page is about.
One timing trap worth knowing: if your server sends noindex and JavaScript removes it after rendering, Google will not reconsider. It dropped the page when it read the initial HTML. Never ship noindex in the source expecting a script to undo it. Meta robots tags covers the full directive list.
Frequently asked questions
Do I need index, follow on every page?
No. Both are the default, so a tag saying index, follow changes nothing at all. SEO plugins output it by the million because it looks reassuring in a settings panel. It is harmless and pointless; leaving it out has exactly the same effect.
What is the difference between the meta tag and X-Robots-Tag?
Nothing, for an HTML page. The header exists for files with no head to put a tag in, which in practice means PDFs, images and generated downloads. If you need to keep a PDF out of search, the header is your only option.
I added noindex but the page is still in Google. Why?
Almost always because the URL is also blocked in robots.txt. A blocked page is never fetched, so the noindex inside it is never read, and the URL can stay indexed indefinitely. Unblock the page so the crawler can get in and read the tag.
Does noarchive still do anything?
Very little on Google, which retired the cached-page link in 2024. Bing still honours it. It is rarely worth setting now.
Can I use this to hide a private page?
No. Noindex keeps a page out of search results, but the page stays publicly reachable by anyone with the URL. Anything genuinely private needs a password or a login, not a tag.