SEO for AI engines (GEO)

What is GEO, and which part of it is actually proven?

GEO is short for generative engine optimization, the name for work aimed at being visible inside the generated answers of search engines and assistants. AEO is the same thing under another name.

Google has an official position on both words, and reading it first will save you time: in its guide to optimizing for generative AI features it writes that from Google Search's perspective, optimizing for generative AI search is optimizing for the search experience and thus still SEO, and it adds that if you are considering third-party AEO or GEO advice or services, you should review its guidance on evaluating third-party SEO advice. We unpacked that position in detail in Google said not to build a special file for AI, and we will not repeat it here.

But Google is not the only engine in the world, and this lesson is about exactly the part that post does not cover. ChatGPT, Claude and Perplexity each run their own crawlers and their own policies, and each publishes its own documentation. There are things in there that are documented and certain, and almost nobody in the Iranian market talks about them.

So before any advice, let us drop the claims into four buckets and keep those four buckets to the end of the lesson: what you measure yourself, what the vendor documented, what is a reasonable inference, and what is pure guesswork sold as a package. Every sentence in this lesson sits in one of the four, and we say which one each time.

Which crawlers come to your site, and what does each one do?

This section is the documented part of the lesson and the most valuable one, because it is the only lever actually in your hands and its outcome is binary: either they are allowed in or they are not.

The common mistake is to assume each company runs one crawler. They do not. OpenAI writes in its own documentation that OAI-SearchBot and GPTBot are two independent settings: one for being visible in search, the other for model training. A site that blocks GPTBot and leaves OAI-SearchBot open is out of training and present in search. The same page writes that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links, and that it can take around twenty-four hours from a robots.txt change for their systems to adjust.

Anthropic runs three separate robots and explains all three on its own support page: ClaudeBot, which collects web content that could contribute to model training; Claude-User, for when an individual asks a question and the system goes and fetches your site; and Claude-SearchBot, which works on the quality of search results. That page writes that disabling Claude-User prevents their system from retrieving your content in response to a user query, which may reduce your site's visibility.

And then the Google case, which is misunderstood more than any other. Google-Extended has no separate user agent string at all; Google's own documentation says crawling is done with existing Google user agent strings and that this token acts purely as a control in robots.txt. However long you search your log, you will never find a request calling itself Google-Extended. What the token controls is whether your content may be used for training future generations of the Gemini models and for grounding in those same products, and the same page states plainly that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal.

Read the table once from left to right and notice how simple the decision becomes. If you want to be visible in ChatGPT answers but keep your content out of training, you know exactly which two lines to write. The documentation of all three companies is in this lesson's source list.

What does an assistant usually quote from a page?

Here we move into the third bucket: inference. Let us say plainly that none of these companies has published a list of criteria for choosing a source, so everything in this section is reasoning rather than documentation.

The reasoning is simple. When an assistant builds an answer it needs a sentence it can quote without cutting it up and without adding explanation. A page that answers the question in its first two sentences has such a sentence. A page that answers in the sixth paragraph, after a warm-up, does not. In our experience that single difference does more for being quoted than anything else, and it costs nothing, because it is better for a human reader too.

Three other things help for the same reason: use one term consistently across the page instead of playing with synonyms; link every claim that needs authority to its source, so that if the model wants to check, the path is right there; and lay out the page with proper headings so the relevant part can be lifted out on its own.

Now the point most people leave out, and we say it because it is this site's own policy. Google writes in that same guide that you do not need to write in a specific way to appear in AI features, that there is no requirement to break content into tiny pieces, and that no special schema is needed. So do not sell the four recommendations above as Google's rules. They are an inference about how a language model builds a quotation out of a page, and they are independently right for a human reader. Those two reasons are enough to do them, and there is no third one.

We run this same rule on this Learn section: every lesson is required to answer in its first two sentences, and those two sentences have to make sense alone, without the rest of the page. The first reason is a reader in a hurry; that an assistant can also lift those two sentences is a side effect and not the goal.

Should you build an llms.txt or not?

Short answer: build one if it costs you nothing, but pay nobody for it, and expect no effect on your Google ranking from it.

The document on Google's side is explicit. In that same generative AI guide, in a section it labels mythbusting, it writes that you do not need to create new machine readable files, AI text files, markup or Markdown to appear in Google Search including its generative capabilities, because Google Search itself does not use them. And it immediately adds that it is completely fine if you decide to create and maintain such files, but doing so will neither harm nor help your visibility or rankings in Google Search, as Google Search ignores them.

The other side of the story is real too, and usually left out. llms.txt is a proposal Jeremy Howard published in September 2024, now in its second version at llmstxt.org. That page says thousands of sites publish the file, documentation platforms generate one automatically, Chrome's Lighthouse audits sites for one as part of its agentic browsing checks, and the AI labs themselves publish llms.txt files for their own developer documentation.

So hold both true sentences at once: Google ignores it, and it is at the same time a live convention that other tools do read. We keep one on this site and we are not taking it down, because it costs us one text file that is regenerated automatically. And we do not sell it to anybody, because we have no measured effect to sell.

If somebody is charging you separately to build this file and promising you a result in Google, Google's own documentation has already answered them.

How do you find out whether you are being seen at all?

You have one place of real measurement and one half-measurement, and the rest cannot be measured at all. Let us say honestly which is which.

The real measurement is your own server log. Every request these crawlers make to your site lands there with a name, a time and a response code. That is where you see which of them came, which pages they fetched, and above all what code they got: a page that returns two hundred to your browser can return 403 to a robot, and your server, firewall or CDN can arrange that without telling you. The fast path at the bottom of this page is exactly that check.

The half-measurement is Search Console. Google points in that same guide to the generative AI performance report in Search Console; what that report does and does not show is explained separately in the Search Console AI report.

And the thing nobody can measure: whether your site was quoted inside a private conversation between a user and an assistant. That conversation is available to no tool. This is why Google itself warns, in the mythbusting section, to be wary of third-party tools that promise ranking success or claim to use internal Google metrics, and writes plainly that no third-party tool has access to their internal ranking or AI systems.

There is also a manual method some people call GEO tracking: ask the assistants themselves about your subject and see which sites they point to. It is useful, but it is not measurement; the answer changes each time, it depends on your account history, and the sample size is one. Keep it as a qualitative observation and never make a decision on a chart you built out of it.

So what is actually worth doing today?

In the order that they are worth, rather than the order they are sold in GEO packages.

First, make sure your pages are crawlable and indexable at all. A page that is not in the index is not in any AI answer either. If that part is limping, go to technical SEO and put the rest of this lesson aside for now.

Second, and the most common silent failure we meet in practice: make sure your firewall or CDN is not blocking these robots. Many security services are hard on unknown bots by default, and the result is a site with a completely open robots.txt that in practice returns 403 to OAI-SearchBot. Nobody tells you; you simply never get quoted.

Third, set the tokens deliberately rather than by accident. Look at the table in the second section and decide which ones you want and which you do not. If you do not decide, your server default and your security plugin decide for you.

Fourth, answer in the first two sentences and keep one term stable. That is inference and not law, but it is right for the reader too, so it costs you nothing.

And the thing not worth doing: buying a package called GEO that promises placement in AI answers. Nobody outside these companies has access to their systems, and nobody can guarantee a position they cannot see. If you want the four things above done on a real site, that is what our SEO service is; but if you have a technical team, all four are yours to do, and we would rather you did.

One prediction, which may turn out wrong and which we would rather you remembered we made: until one of these companies publishes a report of its citations, everything sold as GEO measurement is a reconstruction of a guess wearing the clothes of a chart. If such a report ever appears, we will rewrite this lesson that day and the verified date at the top of the page will change.