Generate a valid robots.txt file to control how search engines crawl your site. Free, no signup required. Use it right in your browser.
What it does, limits, use cases, tips, and answers to common questions.
robots.txt tells well behaved crawlers which paths they may request. It is not a security wall. Private URLs still need authentication.
This generator writes a robots.txt from allow/disallow rules and a sitemap URL. Disallowing /admin is fine. Disallowing / because you are embarrassed about thin pages is a blunt tool. Noindex on those pages is usually cleaner.
Syntax errors (missing colons, wrong user agent) make the file ignored. Test after deploy.
Bad bots ignore robots.txt.
Disallow does not remove a URL already indexed. Use noindex.
This does not replace authentication.
Blocking staging query params.
Pointing crawlers at sitemap.xml.
Keeping search result URLs out of crawl budget.
Never Disallow: / on a site you want AdSense to review.
Not by itself. Use noindex or login.
Some honor extra user agents; many do not.