Free Robots.txt Generator
Build a robots.txt with sensible defaults. Block or allow AI crawlers (GPTBot, ClaudeBot, PerplexityBot), set per-bot rules, and add your sitemap — all in one place.
Block AI / LLM crawlers
Toggle the bots you want to block from training on or scraping your content. Most sites should allow GPTBot, ClaudeBot, and PerplexityBot for AI-search visibility (AEO).
Custom per-bot rules
Advanced: add specific rules for individual bots.
robots.txt
# Generated by SEOSpectator robots.txt generator # https://seospectator.com/tools/robots-txt-generator User-agent: * Allow: / Disallow: /admin/ Disallow: /private/ Disallow: /checkout/ Disallow: /cart/ Sitemap: https://example.com/sitemap.xml
Upload as robots.txt at the root of your domain. Validate in Google Search Console.
What robots.txt does (and doesn't do)
robots.txtis a polite request. It tells well-behaved crawlers (Googlebot, Bingbot, most AI bots) which paths they shouldn't fetch. It is not a security mechanism — malicious bots ignore it, and disallowed pages can still be indexed if linked from elsewhere. For real privacy, use authentication or noindex meta tags.
The AI bot question
Since 2023, a wave of AI training crawlers has appeared: GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), Google-Extended (Google for Bard/Gemini training), CCBot (Common Crawl), and others. You have a real choice:
- Allow them — your content gets cited in AI answers. ChatGPT, Claude, Perplexity, and Google AI Overviews can surface your brand to users who never type a search query.
- Block them — protect proprietary content from being used as training data. Useful for paywalled, premium, or rights-sensitive material.
- Mix — block training crawlers but allow inference bots that cite sources. Common pattern for publishers.
Where to put robots.txt
At the root of your domain: https://yourdomain.com/robots.txt. Not in a subdirectory. Each subdomain needs its own file. After publishing, validate it in Google Search Console under Settings → Crawling.
One tool was useful. Want all 42 checks?
Run the full SEOSpectator audit on any URL — every check, with the AI Coach explaining what to fix and how. Free to try, no credit card.