robots.txt Generator

Build a robots.txt for your site.

Mode

Paths to block (one per line)

Sitemap URL

Crawl-delay (s)

robots.txt

User-agent: *
Disallow: /admin/
Disallow: /api/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

Save as robots.txt at the root of your domain. Honored by well-behaved bots (Google, Bing, DuckDuckGo). Bad-actor scrapers ignore it. Use server-level auth for actual access control.

About

Pick allow-all, block-all, block specific paths, or whitelist specific bots. Add a sitemap URL and crawl-delay if you need them. Output is ready to save at /robots.txt.

How to use

Pick a mode.
Fill in paths or bots.

FAQ

Will malicious scrapers obey it?+

No. robots.txt is honored by Google, Bing, and other well-behaved crawlers. Bad-actor scrapers ignore it. For real access control, use server-level auth.