BisyriYou have a well-built website. It ranks on Google. Customers find you. But when someone asks ChatGPT or Perplexity for a recommendation in your space, your business doesn't come up — even when you're the best answer.
There's a good chance one line in your robots.txt is the reason.
What Is robots.txt and Why Does It Matter for AI?
Your robots.txt file sits at yourdomain.com/robots.txt. It tells crawlers which parts of your site they can and cannot access. Google has Googlebot. ChatGPT has GPTBot. Perplexity has PerplexityBot. robots.txt rules apply separately to each — and most were written long before AI crawlers existed.
If your robots.txt blocks an AI crawler, that crawler cannot read your content. The AI engine it serves cannot cite you. Full stop.
The 7 AI Crawlers You Need to Know
- GPTBot — ChatGPT's main crawler (OpenAI)
- OAI-SearchBot — OpenAI's search-specific crawler
- PerplexityBot — Perplexity's crawler
- ClaudeBot — Anthropic's content crawler
- Claude-Web — Anthropic's web browsing agent
- Google-Extended — Google's AI and Gemini training crawler
- CCBot — CommonCrawl, used as training data by many AI systems
If any of these appear under a Disallow: / rule in your robots.txt without a corresponding Allow: /, that crawler is blocked.
How to Check Your Own Site Right Now
Open your browser and go to yourdomain.com/robots.txt. You're looking for two patterns:
A blanket block — blocks everything including AI crawlers unless specific agents have Allow rules:
User-agent: *
Disallow: /A named block — specifically targets one AI crawler:
User-agent: GPTBot
Disallow: /Either of these, without a correction, means AI engines cannot read your site.
What a Correct robots.txt Looks Like
If you want all AI crawlers allowed, your robots.txt should have no blocking rules for them, or include explicit allows:
User-agent: GPTBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /What I Found Most Commonly
[BISYRI: Insert your own finding — e.g., "Of the sites I analyzed, about X% had a blanket Disallow: / rule under User-agent: * that was blocking all AI crawlers, or had specifically named GPTBot or Google-Extended in a Disallow block. In most cases the rule had been added years ago for a reason that no longer applied."]
The most common culprit: WordPress sites with a plugin-generated robots.txt that blocked all bots except Googlebot. This was a reasonable SEO practice years ago to prevent duplicate content crawling — it's now an AEO liability.
The Fix
If you're on WordPress, check your SEO plugin's robots.txt editor (Yoast, Rank Math, or similar) and add explicit allows for the AI crawlers you want to permit. If you manage robots.txt directly, edit the file and redeploy. Then visit yourdomain.com/robots.txt to confirm the changes are live.
This is a 5-minute fix. The impact: AI engines can crawl and potentially cite your content from the next time they index your site.
Also Consider: llms.txt
Beyond robots.txt, there's an emerging convention: a /llms.txt file that gives AI crawlers curated context about your site — brand positioning, key pages, what you want to be known for. It's optional and not yet widely adopted, but early movers get the benefit of clearer AI entity recognition. See llmstxt.org for the format.
Check All 7 AI Crawlers at Once
The free AEO Analyzer fetches your robots.txt and checks all 7 major AI crawlers automatically — showing you which are allowed, which are blocked, and whether llms.txt is present. Takes about 8 seconds. No login required.