Search Engine Access File (robots.txt)
FreeCreate the file that tells Google and AI which pages they can and can't visit.
User-agent: *
Disallow:Save this as robots.txt in your site's (e.g. example.com/robots.txt).
Free to use — 3 exports a day. Upgrade for unlimited.
Search Engine Access File (robots.txt) — FAQ
What is a robots.txt file?+
robots.txt is a plain-text file at the root of your domain that tells search engine crawlers which parts of your site they may or may not crawl. It's a core part of technical SEO.
Should I block crawlers from any pages?+
Commonly you block admin areas, carts, checkout, and internal search results — not content you want indexed. This tool includes safe presets, including a recommended WordPress configuration.
Do I need to add my sitemap to robots.txt?+
It's a best practice. Adding a Sitemap: directive helps search engines discover all your URLs faster. Just paste your sitemap URL and it's included automatically.
Related free tools
How Do You Show Up on Google & AI?
Type your business name — and your website if you have one — and see how you currently appear on Google, plus whether AI assistants can even read your site. Real checks, no signup.
OpenLocal AI Visibility Grid
See whether AI recommends your business for 'best [service] in [city]' across any cities worldwide — and in any language.
OpenAI Visibility Checker
See if AI assistants like ChatGPT recommend your business — and which competitors they name instead.
OpenLocal Listings & Google Profile Optimizer
Get optimized Google Business Profile content and the exact directories to be listed in — for any country.
OpenStart growing your website today
Create your free site in minutes — no credit card, and you own and export everything you build.