SEO Tools
Robots.txt Generator
Build a valid robots.txt file with allow, disallow and sitemap directives — no syntax mistakes, ready to upload to your site root.
- Allow-all, block-all or custom rules
- Per-path disallow directives
- Automatic Sitemap declaration
- Valid, copy-ready robots.txt output
Local processing. Everything you type stays in this browser tab — nothing is sent to a server, and no account is required.
About this robots.txt file
- Allow everything writes an explicit
Allow: /line — the safe default for most sites. - robots.txt prevents crawling, not indexing. Use a no-index directive when a page must stay out of the results entirely.
-
Save the file as
robots.txtat your domain root so it answers at https://example.com/robots.txt.
How it works
robots.txt sits at the root of your domain and tells crawlers which parts of the site they may read. It is a small file with large consequences: one wrong line can deindex an entire site, and blocking your own CSS or JavaScript can stop pages rendering properly in the results. This generator builds the file from simple choices so the syntax is always valid.
Using the tool
- Pick the mode: allow everything (the normal case), block everything, or custom rules only.
- List the paths to disallow, one per line — each line already starts with a slash.
- Add your sitemap URL so crawlers can discover your pages without relying on links alone.
- Copy the output and save it as robots.txt in the root directory of your site.
Everything runs locally in this browser tab. See the full list of SEO tools for more utilities, or the general toolbox for HTML, JSON and text tools.
Frequently asked questions
- Does robots.txt stop a page being indexed?
- Not reliably. It prevents crawling, but a URL blocked by robots.txt can still appear in results if other sites link to it. For a true no-index you need the meta robots or X-Robots-Tag directive on the page itself.
- Should I block my whole site?
- Only during a launch or a migration where the site must stay private. Remember that blocking also stops Googlebot from seeing any updates — remove the block as soon as you go live.
- Where does robots.txt go?
- At the very root of the domain, for example https://example.com/robots.txt, served with a 200 response — not behind a redirect.