Free tool · SEO · No sign-up
Robots.txt generator
Build a correct robots.txt with your sitemap, blocked folders and AI crawler choices, and catch the mistakes that hide sites from Google.
robots.txt
Runs in your browser. Nothing you type is sent to us or stored.
How to use it
- 01
Start from a preset
Allow everything (most small sites), or the WordPress preset that keeps the admin area out.
- 02
Add your sitemap and blocks
Your sitemap address, any folders search engines shouldn't crawl, and whether AI crawlers may read your site.
- 03
Download and upload
Save robots.txt to the main folder of your site, so it opens at yoursite.com/robots.txt.
Questions about this tool
What does robots.txt do?
It tells search engine and other crawlers which parts of your site they may crawl. It's a request that well-behaved crawlers follow, not a security lock.
Can robots.txt remove a page from Google?
No. Blocking a page in robots.txt stops Google from reading it, but the address can still show in results. To keep a page out of Google, let it be crawled and add a noindex tag instead.
Should I block AI crawlers?
It's your choice. Blocking them asks AI companies not to use your pages for training or answers; allowing them may help your business get mentioned in AI answers. Blocking Google-Extended doesn't affect your Google Search rankings.
I use Shopify or Wix. Can I use this?
Those platforms create robots.txt for you. Shopify lets you edit it through a theme template (robots.txt.liquid); most small stores are fine with the default.
More free tools
All free tools- Readability checkerScore how easy your text is to read, and find the long sentences to cut.Open tool
- URL slug generatorTurn page titles into clean, short URLs, one at a time or in bulk.Open tool
- 301 redirect generatorTurn a list of old and new URLs into redirect rules for .htaccess, Nginx, Vercel or Netlify.Open tool
Want us to handle it? See our technical seo services.
Tell us what you need. We reply with where we'd start and a quote.