Crawler Rules
Create allow and disallow directives for any user-agent.
Robots.txt Generator
Build a robots.txt file with user-agent, allow, disallow, sitemap, copy, and download controls.
Crawler RulesSitemap LineDownload File
Crawler rules
User-agent, sitemap, and allow/disallow lines.
robots.txt
4 lines
User-agent: * Allow: / Disallow: /api/ Sitemap: https://aidengpt.com/sitemap.xml
Capabilities
What this tool handles
Use Robots.txt Generator to check the input, tune the settings, and copy a clean output.
Copy Output
Copy the generated robots.txt content instantly.
Download File
Download a ready-to-upload robots.txt file.
Workflow
Run it in three steps
Create crawler rules and download a robots.txt file.
1
Set User-agent
Choose the crawler target, such as * for all bots.
2
Add Rules
Create allow and disallow directives for paths.
3
Copy or Download
Copy the output or download robots.txt.
Good for
Common jobs this tool supports
Crawler RulesBuild rules for user-agents
Sitemap LineAdd sitemap location easily
Copy OutputCopy the file content
Download FileSave a robots.txt file
FAQ
Answers before you run it
Practical notes about output, privacy, and common settings.
No. Robots.txt controls crawling, not guaranteed indexing removal. Use noindex or removal tools for index control.
Use next