About CrawlKit

crawlkit.online is a small set of free tools for people who look after websites. No account, no email, no trial that runs out. You open a page, use the tool, close the tab.

Who makes it

My name is Faaiz Mahmood. I am a software engineer with a bachelor's degree in computer science, and I work on full-stack projects and as an AWS cloud engineer. I write the code, the guides and the documentation on this site, and I maintain it.

I am not an SEO consultant, and I would rather say so than imply otherwise. What this work needs is not marketing expertise but the ability to read a specification and implement it precisely, which is the job I do every day.

Why these tools exist

The rules behind robots.txt, sitemaps and crawl directives are not folklore. They are written down, publicly, in exact language: RFC 9309 defines robots.txt parsing and matching, the sitemaps protocol defines what belongs in a sitemap, and Google documents its own behaviour in detail.

A surprising number of free tools built on those specifications get the details wrong. They treat robots.txt rules as first-match when Google uses longest-match. They count characters when Google measures pixels. They emit rewrite rules with a leading slash that will not match inside an .htaccess file. The mistakes are small and the consequences are not, because someone trusts the output and ships it.

So each tool here is written against the specification, and each one publishes the rules it follows. Where crawlers genuinely disagree, or where Google's own documentation calls a case undefined, the tool says so instead of inventing a confident answer. You should be able to check the reasoning rather than trust the tool.

Nothing you type leaves your browser

Every tool runs as JavaScript on your own machine. The robots.txt you paste, the URLs you test, the rules you generate: none of it is transmitted, logged or stored, because none of it reaches a server at all.

That is a deliberate trade. It is why the tools ask you to paste a file rather than fetching your site for you, which would be more convenient and would mean sending your domain somewhere. It also means the tools work on a staging site behind a login, which is often exactly what you need to test and exactly what a fetch-based tool cannot reach.

It does impose real limits. A browser cannot crawl your site, follow your redirects to see where they land, or read a live page. Where a tool cannot do something for that reason, its page says so plainly.

What is here

The tools page lists everything, and the guides explain the rules each tool applies in plain English. The guides are not filler around the tools; they are where the reasoning lives, so you can disagree with it.

More tools are in progress. They ship when they are correct rather than on a schedule.

How the site is paid for

The site is free and supported by advertising. Ads never change what a tool produces, and there is no paid tier, no upsell and no data to sell, because the tools never send me your data in the first place. The privacy policy sets out exactly what is and is not collected.

Corrections and contact

If a tool gives a wrong answer, that is a bug worth knowing about, and I would rather hear it than not. The same goes for anything inaccurate in the guides. Use the contact page or email contact@crawlkit.online.