Meta Robots Tags: A Plain-English Reference

By Faaiz Mahmood · Updated 11 September 2026 · 8 min read

Robots.txt works at the level of paths and whole folders. The meta robots tag works at the level of a single page, and it controls something robots.txt cannot touch: whether the page is allowed into the index, and what the result looks like when it gets there.

1. The two ways to set it

In HTML, a tag in the <head>:

<meta name="robots" content="noindex, follow">

As an HTTP response header, which works for any file type:

X-Robots-Tag: noindex, follow

They are equivalent for HTML. The header is the only option for things with no head to write into, which in practice means PDFs, images, and generated downloads. In Apache you would scope it with a file match:

<FilesMatch "\.pdf$">
    Header set X-Robots-Tag "noindex"
</FilesMatch>

If both are present and they conflict, the more restrictive instruction wins. Do not set both.

2. Targeting one crawler

name="robots" addresses every crawler that reads the tag. Swap in a specific user-agent token to talk to one:

<meta name="googlebot" content="noindex">
<meta name="bingbot" content="noarchive">

A crawler-specific tag overrides the generic one for that crawler, and the generic one still applies to everyone else. This is worth knowing mostly so you recognise it when you inherit a site. Genuine reasons to differentiate are rare.

3. The directives worth knowing

Combine with commas. Order does not matter, and values are case-insensitive:

<meta name="robots" content="noindex, follow, noarchive">

4. The combinations that actually get used

noindex, follow is the workhorse. Keep the page out of results, but keep following its links so the pages it points to still get discovered and still receive value. This is what you want on paginated archives, filtered listings and internal search pages that link to real content.

noindex, nofollow is heavier than it looks. It cuts the page off completely. Reasonable for a thank-you page or a checkout step that links nowhere useful. A poor choice for anything that links to pages you care about.

max-snippet:-1, max-image-preview:large tells Google it may show a full description and a large image. Publishers use it to get richer-looking results. It is permissive rather than restrictive, and it is one of the few cases where adding directives can help rather than merely restrict.

index, follow does nothing at all. It is the default. SEO plugins output it by the million because it looks reassuring in a settings panel. Harmless, pointless.

5. The mistake that undoes all of it

None of these tags work on a page that Googlebot is not allowed to fetch. If the URL is blocked in robots.txt, the crawler never loads the HTML, never sees the tag, and the instruction is simply never received.

This is the most common failure in the whole area, and the fix is counter-intuitive: unblock the page so the crawler can read the noindex. It is covered in full in noindex vs disallow, which is worth reading before you touch either file.

The same applies to the HTTP header. It arrives with the response, so no response means no header.

6. JavaScript and timing

Google renders JavaScript, so a robots tag injected by script can be picked up. Relying on that is risky in one specific direction.

If the HTML ships with no robots tag and JavaScript adds noindex after render, there is a window where Google saw an indexable page. It generally resolves on rendering, but it is an avoidable race.

The reverse is worse and is worth stating clearly: if the server sends noindex and JavaScript tries to remove it later, Google will not reconsider. It dropped the page when it read the initial HTML. Never ship noindex in the source expecting script to undo it.

Put the tag in the server-rendered HTML whenever you can.

7. Where to use which

Note the filtered-listing row. A filtered page is usually a duplicate with a sensible parent, so a canonical tag is better than noindex: it keeps the ranking value and moves it, where noindex throws it away.

8. Checking it

View source and search for name="robots". Check there is exactly one. Two tags with different values is a common WordPress problem, where the theme emits one and a plugin emits another.

For the header version, look at the response headers directly:

curl -I https://example.com/report.pdf

Then confirm with URL Inspection in Search Console. It reports indexing allowed or not allowed as a separate line from crawling allowed, which is exactly the distinction that trips people up.