Robots.txt Generator
Generate a valid robots.txt file with custom disallow rules and a sitemap reference.
A robots.txt file is the first thing well-behaved search engine crawlers check before indexing a site, telling them which paths they're welcome to crawl and which to skip — admin panels, internal search result pages, API routes, or duplicate content paths that would otherwise waste crawl budget or leak into search results. Getting the syntax wrong (a single misplaced rule) can accidentally block an entire site from being indexed, which is a surprisingly common and costly mistake.
This generator produces a syntactically correct robots.txt from a simple list of paths to disallow, plus your sitemap URL, formatted exactly as major search engines expect. The output also includes an explicit Allow: / rule for the root, since some crawlers behave more predictably with an explicit allow alongside disallow rules.
Remember that robots.txt is a voluntary convention, not an access control mechanism — well-behaved crawlers respect it, but it doesn't prevent a URL from being accessed directly, and it doesn't reliably keep a page out of search results if other sites link to it (use a noindex meta tag for that instead).
How to use Robots.txt Generator
- 1
List paths to disallow
Enter one path per line that you want to block crawlers from (e.g. /admin, /api).
- 2
Add your sitemap URL
Enter the full URL to your sitemap.xml file.
- 3
Copy the generated file
Copy the output and save it as robots.txt at your site's root.
Features
- Generates syntactically correct robots.txt output
- Includes your sitemap reference automatically
- Simple one-path-per-line input
- Instant live generation as you type
Frequently asked questions
Common mistakes to avoid
- Accidentally disallowing the entire site with a bare 'Disallow: /' rule under 'User-agent: *'.
- Using robots.txt to try to hide sensitive content instead of proper access control — it's a publicly readable file and a poor security mechanism.
- Forgetting to update robots.txt after restructuring a site's URLs, leaving stale rules that block nothing useful.
Get new tools in your inbox
One email a month with new tools, guides, and updates. No spam, unsubscribe anytime.