Your robots.txt Isn't Why ChatGPT Ignores You

One of the first things every AI SEO checklist tells you: make sure your robots.txt allows GPTBot. Check that you haven't blocked PerplexityBot or ClaudeBot. The assumption is that AI systems crawl your website directly and blocking them means they can't see you.

We tested this across 1,000 businesses. robots.txt configuration showed a -7% correlation with ChatGPT visibility. Negative. Businesses that explicitly allowed AI crawlers were slightly less likely to appear in ChatGPT results than businesses that didn't bother.

This makes sense once you understand how ChatGPT finds local businesses. It doesn't crawl your site. It doesn't send GPTBot to read your pages. Here's what actually happens:

1. You ask ChatGPT for a plumber
2. ChatGPT's web search tool sends a query to Bing
3. Bing returns its search results
4. ChatGPT reads those Bing results and writes an answer

GPTBot is OpenAI's training data crawler. It collects content to train future models. Blocking it means your content won't be in ChatGPT's training data. But we confirmed that ChatGPT's local recommendations come from live Bing web search, not training data. We tested the same queries with and without web search enabled -- the overlap was near zero. Without web search, ChatGPT hallucinated nearly every business name. With web search, it pulled real businesses from Bing results.

So blocking GPTBot doesn't stop ChatGPT from recommending you. And allowing GPTBot doesn't make ChatGPT recommend you. The crawl permission and the recommendation system are completely separate.

The -7% correlation likely exists because businesses that carefully configure robots.txt tend to be more technically sophisticated. They're the ones with SEO consultants managing their sites. But that technical sophistication isn't translating into ChatGPT visibility -- because robots.txt isn't part of how ChatGPT selects businesses.

We found the same pattern with llms.txt, a newer file format designed specifically to help AI systems understand your site. Overall correlation with ChatGPT visibility: +6%. For plumbers specifically it was +13%, but across other categories it was negligible. It's not doing what people think it's doing.

If you've paid someone to audit your robots.txt for AI search, you didn't get bad advice exactly. It's just irrelevant advice. The file that matters isn't on your server. It's on Bing. And the way to influence what Bing shows is to exist on more platforms, have more signals pointing at your business, and have a website that clearly says what you do and where you do it.

That's less satisfying than editing a text file. But it's what the data says.