The Right Robots.txt for a WordPress Site
The short version
- WordPress serves a virtual robots.txt that is fine for most sites.
- Add your sitemap line. That is the one change nearly every site benefits from.
- Never block
/wp-content/or/wp-includes/. It breaks how Google renders your pages. - Thin archive pages want
noindex, not a disallow rule.
Search for a WordPress robots.txt template and you will find long files blocking a dozen directories, most of them copied from advice that was questionable in 2012 and is actively harmful now. The honest answer is that WordPress ships with something close to correct, and the useful edits number about two.
1. The virtual file
If you have never created one, WordPress generates a robots.txt on the fly. Visit yourdomain.com/robots.txt and you will see roughly:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpThis is sensible. It keeps crawlers out of the admin area while leaving open the one file inside it that themes and plugins call for front-end functionality. Block admin-ajax.php and features on your public pages can break for Googlebot specifically.
The file is virtual: there is nothing on disk. The moment you upload a real robots.txt to your web root, it takes over completely and the defaults are gone, which means you must include the admin rules yourself.
If Settings, Reading, Discourage search engines is ticked, WordPress serves Disallow: / instead and blocks everything. That setting gets left on after launch more often than anyone would like. If a new site is not being indexed at all, check it before anything else.
2. What most sites should have
For a typical blog or business site:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xmlThat is the whole file. The defaults plus a sitemap line.
Get the sitemap URL right for your setup. Core WordPress since 5.5 serves /wp-sitemap.xml. Yoast uses /sitemap_index.xml. Rank Math uses /sitemap_index.xml. All in One SEO uses /sitemap.xml. Open yours in a browser and confirm it loads before putting it in the file. The robots.txt generator will format this correctly if you paste your sitemap URL in.
3. What to add if you have the problem
Only add these if they apply to you. An empty rule costs nothing but it also does nothing, and every line is one more thing to get wrong.
Internal search results. WordPress search URLs are infinite by definition, and there is no reason for a crawler to be generating queries:
Disallow: /?s=
Disallow: /search/WooCommerce cart and checkout. Not search pages in any sense, and they generate session parameters:
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*add-to-cart=Product filters. On a large shop, faceted URLs multiply fast. Block the specific parameters you actually use:
Disallow: /*?filter_
Disallow: /*?orderby=
Disallow: /*?min_price=One warning that applies to all of these. If those URLs are already in Google, blocking them now freezes them there, because the crawler can no longer fetch the page to read a noindex tag. Remove them first with noindex, wait for them to drop, then block. Noindex vs disallow explains why in full.
4. What not to block
This is the part where old templates do damage.
Never block /wp-content/. Your CSS, JavaScript, images and theme files all live there. Google renders pages like a browser, so blocking those assets means it sees an unstyled, broken version of your site. It will report the page as not mobile friendly and judge the layout accordingly. This single rule appears in countless copy-paste templates and it is the most damaging line you can add.
Never block /wp-includes/. Same reason. Core scripts like jQuery are served from there.
Do not block /wp-content/uploads/ unless you have a specific reason. That is where your images are, and blocking it removes you from image search.
Do not block feeds by reflex. /feed/ is harmless and useful.
Do not list private folders. Robots.txt is public, so Disallow: /secret-backups/ advertises exactly what you hoped to hide. Anything genuinely private needs a password.
Do not use Noindex: in robots.txt. Google stopped honouring that unofficial directive on 1 September 2019. Lines like it are ignored today, and anyone relying on them has pages indexed they believe are hidden.
5. Archive pages want noindex, not disallow
WordPress generates a lot of archive pages automatically: category, tag, author, date. On a small site, most add nothing. A tag page holding two posts and no writing of its own is exactly the thin content people try to block.
The instinct is a disallow rule. That is the wrong tool. Blocking the crawl leaves any already-indexed archive stuck in results, and cuts off the links those archives pass to your actual posts.
Use noindex, follow instead, which keeps them out of results while letting link value flow through to the posts. Every major SEO plugin exposes this as a toggle. In Yoast it is under Search Appearance, in Rank Math under Titles and Meta. Date archives and author archives on a single-author blog are usually the first two to turn off. Meta robots tags covers the directives.
6. Editing the file
Three options, in order of preference.
Your SEO plugin. Yoast has a robots.txt editor under Tools, File editor. Rank Math has one under General Settings, Edit robots.txt. These write to the virtual file, which keeps everything in one place and avoids a stale file on disk.
A real file. Upload robots.txt to your web root, the folder containing wp-config.php. It overrides the virtual one entirely, so include the wp-admin rules yourself. This is the most predictable option if you are comfortable with file access.
A filter in your theme. Possible with the robots_txt hook, and rarely worth it. It lives in code nobody will think to check a year from now.
7. Checking it
Open yourdomain.com/robots.txt in a private window. That is what crawlers see, and it is not always what your plugin panel shows, particularly with page caching in front of the site.
Then, in Search Console, open the robots.txt report to confirm Google fetched it successfully and parsed every line. Use URL Inspection on a few important pages and check it says crawling is allowed.
Finally, one check people skip: inspect your home page and look at the rendered screenshot Google shows. If it comes back unstyled, you have blocked an asset directory, and you have just found the most valuable thing on this page.