☎ +40 754 308 781 ✉ contact@goai.ro
f ig tt X

Home › SEO Guide › The files AI agents read

The files AI agents read

The files AI agents read

The question usually comes in the form “what file do I need to put on my site to show up in ChatGPT?”. The short answer is that the most promoted file does almost nothing, while the one that really matters has been on your site for twenty years and is called robots.txt.

This chapter is the most perishable in the whole guide. The data below was checked at the source on 18 August 2026, directly in each provider's documentation. If you are reading it much later, check it again.

Who actually reads your site

There is no such thing as “an AI bot”. Each company uses several bots, with different purposes, and the distinction between them is the most important business decision in this chapter.

BotWhoWhat it does
GPTBotOpenAICollects content that can be used for training models
OAI-SearchBotOpenAIIndexes for ChatGPT search. This one can bring you citations
ChatGPT-UserOpenAIOpens a page because a person asked for it in a conversation
OAI-AdsBotOpenAIChecks the safety of pages submitted as ads
ClaudeBotAnthropicCollects content for training
Claude-SearchBotAnthropicImproves search result quality
Claude-UserAnthropicOpens pages at a user's request
Google-ExtendedGoogleControls use for training Google's models. Does not affect your position in Search
PerplexityBotPerplexityIndexes for Perplexity answers
CCBotCommon CrawlPublic archive, then used by a great many others

The distinction that matters: blocking training is not the same thing as blocking citation.

If you block GPTBot, your content does not go into the training of future models, but you can still be cited in answers through OAI-SearchBot. If you block everything, you disappear from both. They are two separate decisions, and most companies mistakenly make them as one.

robots.txt, the only file that really matters

It lives at your-site.ro/robots.txt. Open it now, it is public. Each block tells a bot what it is allowed to read.

For a company that wants to be found, the reasonable configuration allows the search bots and leaves the decision about training up to you. The “I want to be cited, I don't care about training” version looks like this:

# OpenAI
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

# Anthropic
User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

# Perplexity
User-agent: PerplexityBot
Allow: /

If you also want to stay out of training, add blocks with Disallow: / for GPTBot, ClaudeBot, Google-Extended and CCBot. Keep one thing in mind, though: Google-Extended has nothing to do with your position in Google Search. Blocking it does not remove you from the results.

A practical warning: not all bots respect the file. The big companies publicly state that they do. For the rest, robots.txt is a request, not a fence.

llms.txt: what the sources say, not what gets written in articles

Since 2024, a proposal has been circulating for an llms.txt file, a summary of the site written for language models. The idea is good on paper. The reality, checked in August 2026, is less enthusiastic.

Google is explicit in its own documentation:

You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them.

Google Search Central, optimization guide for generative AI features, accessed 18 August 2026

Google adds that files of this kind neither help nor hurt rankings, because they are simply ignored, and that it is fine to keep them if other systems use them. On the same page, Google explicitly discourages what it calls AEO or GEO tricks, among them creating text files for AI.

None of the big providers, OpenAI, Anthropic, Google or Perplexity, documents that its production system reads llms.txt to decide what to answer a user.

Where it is useful nonetheless: in tools for programmers. Coding assistants read documentation in real time, and a well-made llms.txt sends them straight to the right pages. If you sell a technical product with documentation, the file makes sense. If you run a guesthouse or a dental practice, it will not bring you a single client.

Our position, to be clear: we have llms.txt on our own site and we are keeping it, because it costs nothing and because we work with tools that use it. But we do not sell it to anyone as an AI visibility tool, because we have no way to prove that it works that way.

The trap of files written for machines

The real problem with files of this kind is not that they don't help. It is that nobody looks at them again after putting them up.

On our own site, llms.txt published the details of an old company for months on end: a different tax code, a different founding year, a different legal form from the real one. The site's pages were correct. The file, which no visitor opens, had been left unchanged from an earlier version of the company.

A person would have noticed immediately. But nobody reads a file meant for machines, and it kept telling any potential automated reader exactly the things we had corrected everywhere else.

Hence the rule: any file you create for machines goes on the list of things to check at every change of company details, alongside the footer and the legal pages. A file for machines that has been left out of date is worse than one that is missing, because a missing file contradicts nothing.

What is actually worth doing instead

The same sources that say special files are not necessary also describe what matters: useful content, a structure a bot can crawl, verifiable identification details, pages that load. In other words, exactly what the other chapters of the guide describe.

One thing is worth adding nonetheless, because it is cheap and actually gets read: prices published in a simple form. An agent comparing providers for a person cannot read a price hidden behind a contact form. If your services have fixed prices, a clear page with them puts you in the comparison. If they don't, say plainly what the price depends on; that too is information an assistant can relay correctly.

What changed in June 2026

Until now, the question “do I show up in generated answers?” could only be checked manually. Since 3 June 2026, Google has started showing in Search Console a performance report for generative AI features, separately for Search and for Discover.

The report shows how many times the site's URLs appeared in generative features, such as AI Overviews and AI Mode, and which pages exactly appeared. It can be broken down by device and by country.

Two limitations to keep in mind. First: Google started the rollout with a subset of sites, as a test, so you may not have it yet. Second: the report counts impressions, not visits, and in the chapter on verification we explained why impressions alone say little. Still, it is the first official measurement we have, in place of manual checks.

To do, in this order

  • Open your-site.ro/robots.txt and read it. Many sites block AI bots without the owner knowing, through settings put in place by a theme or a plugin.
  • Decide the two things separately: do you want to be cited in answers, and do you want to go into training. They are not the same question.
  • If you want citations, check that OAI-SearchBot, Claude-SearchBot and PerplexityBot are not blocked.
  • Publish your prices in a simple form, or explain plainly what they depend on.
  • If you have llms.txt, put it on the list you check at every change of company details. If you don't have one, it is not urgent.
  • Check in Search Console whether you already have the report for generative features.

Sources, all accessed on 18 August 2026