robots.txt generator
Build a robots.txt file from the crawl rules you choose.
- 1Add input
- 2Adjust settings
- 3Get your result
Tool input and files are processed in this browser without being uploaded.
Before you start
Generate one robots.txt user-agent group with explicit Allow and Disallow paths and an optional sitemap URL. Robots directives are not access control or secret protection.
How to use this tool
- Paste content into “Rules”, or load the example to try it.
- Review User agent, Sitemap URL (optional) before selecting “Generate robots.txt”.
- Review the Text result, then use the available copy or download controls.
Supported inputs and limits
Pasted text is limited to 1 MiB; review the result before using it.
Worked example
Example input
Disallow: /private/ Allow: /public/
Example options
{
"agent": "*"
}Example output
User-agent: * Disallow: /private/ Allow: /public/
When something does not work
Use the documented input format and keep values within the displayed limits. Review the result before using it in another document or application. Editing the input marks the previous result as stale; a failed run clears it.
Frequently asked questions
Are my files uploaded?
No. Files and tool input are processed in your browser. Moving around the site can keep a temporary workspace in the same tab. Refreshing or closing the tab clears it, so download results you want to keep.
What should I check in this result?
Generate one robots.txt user-agent group with explicit Allow and Disallow paths and an optional sitemap URL. Robots directives are not access control or secret protection.
Documentation & further reading
Related tools
QR code maker
Turn a link or a little text into a QR code.
UTM link builder
Add your campaign details without the guesswork.
Meta tags generator
Fill in page details and copy the resulting meta tags.
Sitemap generator
Turn a list of page URLs into a sitemap XML file.