Robots.txt Generator
Create a valid robots.txt in seconds: allow or block all crawlers, or set per-agent Disallow rules and a Sitemap link. Copy or download the file.
robots.txt
Frequently asked questions
What does robots.txt do?
It tells search engine crawlers which parts of your site they may visit. Place the file at the root: example.com/robots.txt.
Does Disallow guarantee privacy?
No — robots.txt is a polite convention, not access control. Protect truly private areas with authentication.
Why add a Sitemap line?
It points crawlers straight to your sitemap.xml, speeding up the discovery and indexing of pages.