Robots.txt Generator
Compose a valid robots.txt from allow/disallow rules and your sitemap URL.
Rules
User-agent: * Disallow: /admin
Save this as robots.txt at the root of your domain. An empty Disallow: means crawlers may visit everything. Remember that robots.txt is a request, not a lock — don’t use it to hide sensitive pages. Built entirely in your browser.
How to build a robots.txt
Set the user-agent (use * for all crawlers), then add Allow or Disallow rules with the path each applies to. Optionally paste your sitemap URL. The file builds live below — click Copy and save it as robots.txt at your site's root.
Allowing everything
If you don't add any rules, the generator emits an empty Disallow line, which tells crawlers they're free to visit every page. Add Disallow rules only for the paths you want kept out of search results.
Questions
What does robots.txt actually do?
It tells search-engine crawlers which parts of your site they may or may not request. A Disallow rule asks well-behaved bots to skip a path; an Allow rule re-permits a sub-path. It also lets you point crawlers to your sitemap so they discover your pages faster.
Where do I put the robots.txt file?
At the very root of your domain, reachable at https://yourdomain.com/robots.txt. It only governs the host it sits on, so subdomains need their own file. Copy the generated text into a plain file named exactly robots.txt.
Can robots.txt hide private pages?
No. It's a public request, not a security control — anyone can read your robots.txt, and disallowed URLs can still be linked to or indexed. Use real authentication or noindex headers for anything sensitive.