Robots.txt Generator
Generate a valid robots.txt file with a visual rule builder.
Frequently Asked Questions
What is a robots.txt file?
A robots.txt file is a text file placed at the root of your website that tells web crawlers which pages they can or cannot access. It uses simple directives to control crawler behavior.
How do I use this generator?
Add user-agent blocks for the crawlers you want to control, then add Allow or Disallow rules for each. Include sitemap URLs if desired. The live preview shows the exact file content.
What does User-agent: * mean?
The asterisk (*) is a wildcard that matches all web crawlers. Rules under User-agent: * apply to every bot unless overridden by a more specific user-agent block.
Can I block specific bots?
Yes. Add a separate user-agent block with the bot name (e.g., Googlebot, Bingbot) and define rules specific to that crawler. More specific user-agent blocks take precedence over the wildcard.
Is this tool free?
Yes, ToolDock's robots.txt generator is completely free, requires no signup, and runs entirely in your browser.