Robots.txt Generator
Generate a clean robots.txt file with user-agent, allow/disallow rules, and sitemap URLs.
A robots.txt file controls which pages search engine crawlers are allowed to access on your site. Getting the syntax wrong can accidentally block your most important content from being indexed. This generator creates a valid robots.txt with user-agent rules, allow/disallow paths, and a sitemap reference.
robots.txt output will appear here...
How to Generate a Robots.txt File
- Set the user-agent — specify which search engine bot the rules apply to (* for all bots).
- Add allow and disallow rules — define which paths crawlers can and cannot access.
- Add your sitemap URL — include a reference to your XML sitemap.
- Copy the output — save the generated content as robots.txt in your website's root directory.
Frequently Asked Questions
What is a robots.txt file? ▼
Robots.txt is a plain text file placed in a website's root directory that tells search engine crawlers which pages they are allowed or not allowed to access. It follows the Robots Exclusion Protocol. For example, "Disallow: /admin/" prevents crawlers from indexing your admin pages.
Does robots.txt block pages from being indexed? ▼
Not exactly. Robots.txt prevents crawling, not indexing. If other sites link to a disallowed page, Google may still index the URL (without content). To fully prevent indexing, use the "noindex" meta tag or X-Robots-Tag HTTP header in addition to robots.txt.
Where should I place robots.txt? ▼
Robots.txt must be placed in the root directory of your website: https://example.com/robots.txt. It must be accessible at that exact URL. Placing it in a subdirectory or using a different name will not work.
What is the Sitemap directive in robots.txt? ▼
The Sitemap directive tells crawlers where to find your XML sitemap: "Sitemap: https://example.com/sitemap.xml". This helps search engines discover all your pages efficiently. You can list multiple sitemaps.
Can robots.txt block malicious bots? ▼
No. Robots.txt is advisory — well-behaved crawlers (Google, Bing) respect it, but malicious bots ignore it. For security, never rely on robots.txt to protect sensitive content. Use proper authentication, firewalls, and access controls instead.
Is my data stored? ▼
No. All generation happens in your browser. No data is sent to any server.