Free Robots.txt Generator
Build a correct robots.txt in seconds. Pick your crawlers, set what they may reach, add your sitemap, then copy or download the file.
Tools
Small, focused tools for the jobs that come up when you look after a website. No account, no email, no trial that runs out.
Everything here runs inside your own browser. The robots.txt you paste, the URLs you check, the rules you generate: none of it is sent to us, because none of it leaves your device. That is also why the tools ask you to paste things rather than fetching your site for you. A tool that fetched your pages would have to send them somewhere, and we would rather it did not.
Robots.txt Tester: Check Whether a URL Is Blocked
Paste your robots.txt and a URL to see whether a crawler is allowed in, which rule decided it, and the line that rule sits on. Runs entirely in your browser.
Meta Robots Tag Generator
Build a meta robots tag and the matching X-Robots-Tag header, with warnings for the combinations that cancel each other out or do nothing at all.
Redirect Rule Generator for Apache and Nginx
Paste a list of old and new URLs to get correct .htaccess or Nginx redirect rules, with redirect chains, loops and duplicate sources flagged before you ship them.
SERP Snippet Preview
See how your title and description will look in Google, measured in pixels rather than characters, on desktop and mobile.
XML Sitemap Generator and Validator
Build a valid XML sitemap from a list of URLs, or paste an existing one to check it for malformed XML, a wrong namespace, bad lastmod dates and duplicates.
Schema Markup Generator (JSON-LD)
Build valid JSON-LD structured data for Article, BreadcrumbList, Organization and FAQPage, with the script-breaking characters escaped properly.
Why paste instead of enter a URL?
Most tools like these ask for your domain and fetch the file themselves. That is more convenient, and it means your URLs end up in somebody else's server logs.
These tools do the work in JavaScript on your machine. It costs you one copy and paste. In exchange, nothing about your site, your staging domains or your unreleased URLs is transmitted anywhere. For anyone working on a site that is not public yet, that difference matters.
It does place real limits on what a browser-based tool can do. It cannot crawl your site, follow your redirects to see where they land, or read a live page. Where a tool cannot do something for that reason, its page says so plainly rather than pretending otherwise.
Reading, not just doing
Each tool has a guide behind it explaining the rules it applies, so you can check the reasoning rather than trusting the output. The full set is on the guides page.