How a search engine works
A search engine does not search the live web; it answers from a copy it already read and filed in an index. Ranking therefore means being found first, being read correctly second, and being better than the rest of the index last.
- Lesson 1 of 2
- Beginner
- Free, no signup
From link to result: the four steps of a search engine
-
1
Crawl
The crawler finds the page through links and sitemaps.
-
2
Render
The page runs like in a browser so JavaScript content is seen too.
-
3
Index
The understood version of the page is filed in the index.
-
4
Rank
At query time the index is ordered by relevance, quality and context.
This picture is simplified; behind each step sit dozens of systems whose details only Google knows.
Last checked: Facts and tool names in this lesson are re-checked against their sources on this date.
How does Google learn your page exists?
Google's crawler travels the web along links: from a page it knows to a page it does not. A sitemap is the formal list you hand it, and internal links are the road it takes from a known page to a new one. A page with no link pointing at it and no sitemap entry does not exist as far as the crawler is concerned.
robots.txt enters here, and the most common SEO mistake lives at exactly this point: robots.txt talks about crawling, not about indexing. Block a path in it and Google will not read the content; but if a link to that address exists somewhere else, the address itself may still surface in results with no description. To keep a page out of results you need a noindex tag, and that tag is only read if crawling the page is allowed.
Something said less often: Google does not have one crawler. Its own documentation lists a long set of separate crawlers and fetchers, each doing a different job, from the main search crawler to fetchers that run on the spot when a user asks for something.
Why does Google have to run the page like a browser?
Crawling means fetching a file. Understanding the page is the next job, and for that Google renders it: roughly what your browser does, so that content built by JavaScript is seen too.
The practical point is that rendering costs resources and does not always happen immediately after the crawl. A site whose main text only appears after JavaScript runs is more exposed to being understood late than a site that ships the same text in the initial HTML. That is the technical reason behind the plain advice everyone has heard: put the important text in the page itself.
A simple free test: look at the page source, not at what the browser shows you. Whatever is not there is one step later and one step more fragile for Google.
Once read, where does the page get filed?
The understood version of the page enters the index: a large archive that knows, for every phrase, which pages have talked about it. Being indexed is not the same as ranking; it only means the page is eligible to be shown.
Google makes another decision here that many people miss: among similar pages it picks one as the canonical version. Your canonical tag is a hint in that decision, not an order, and Google's own documentation says plainly that it may choose a different version. That is why Search Console shows states like Alternate page with proper canonical tag: the page has no error and is not indexed either.
The noindex tag gains its meaning at this same stage. It takes effect after the page has been read, not before, and that one sentence separates blocking a path from hiding a page.
From discovered to displayed
-
1
Discovered
The address reached Google through a link or a sitemap.
-
2
Crawled
The page file was actually fetched and robots.txt did not block it.
-
3
Indexed
The understood version sits in the index and no noindex or canonical pushed it out.
-
4
Judged relevant
For one specific query, Google considers this page a showable answer.
The narrowing bands only show the order of the stages and carry no numbers. Each step needs the one before it and none of them guarantees the next.
What actually happens at query time?
When someone searches, Google pulls relevant pages from its index and orders them. Google itself states the frame in its How Search works documentation: relevance to that query, quality and authority, usability, and the searcher's context, meaning language, location and device. The details of the systems are confidential and nobody outside Google knows the exact weight of these factors; anyone who claims to is guessing.
One fact that confuses beginners: a ranking is not a fixed number. The same page can sit in two different places for two people in two cities or on two devices, because search context is part of the calculation. That is why "what is my ranking" is not a complete question until you say where and on what device.
If you want the same picture from the business angle, the article what SEO actually is on this site's blog tells the same chain with commercial examples.
Where this chain usually breaks
In practice, a page nobody sees is stuck at one of four links in the chain, and the order of investigation is the order of the chain itself. One: does any link or sitemap actually reach this page? Two: is robots.txt blocking the crawl? Three: is a noindex tag or a wrong canonical keeping it out of the index? Four: is the page indexed but simply not seen as relevant for the phrase you had in mind?
The fourth case is more common than the first three and more frustrating, because no report shows any error. The page is healthy; Google just knows a better answer for that query. At that point the work moves from technical debugging to understanding search intent and the shape of the results page, which is exactly why the next lesson in this path, the anatomy of a Google results page, exists.
What keeps a page reachable, and what quietly stops it
Keeps it reachable
- An internal link from a page that is itself crawled
- The address present in the sitemap
- The main text inside the initial HTML
- A canonical pointing at the page itself
Quietly stops it
- A path blocked in robots.txt
- A noindex tag left over from the build phase
- A canonical pointing at a different page
- A page whose text is only built by JavaScript
Nothing in the right column produces a Search Console error, which is why these are found late.
AI in this kind of work
AI crawlers take the first step exactly as Google does: they fetch the page. The difference is in the steps after that. Google files the page in an index that shows your address to a user; a model's crawler takes the text for training or for answering in the moment, and it may give the answer without anyone entering your site. Deciding whether your site is open to these crawlers is part of SEO today, not a side debate.
Tools that actually help
- Google Search Console The only place holding the real crawl and index data for your own site. Not AI itself, but every analysis that makes this lesson practical starts from that data.
- Claude Good for making sense of a Search Console message or an unfamiliar tag. Iran is not on Anthropic's supported-countries list and there is no official payment route from Iran; its own page says so with the source.
- Gemini The same use for explaining concepts. Google's own page says the Gemini web app runs in over 230 countries and territories, and Iran is not on that list.
- RGB The access and payment layer for Iran, kept separate from the tools themselves: what actually opens, and how it gets paid for.
Where it backfires
A language model will confidently invent a story about "why my page was not indexed": it cannot see your report, and nobody outside Google knows those systems in detail. Read the Search Console report first, then ask the model to explain concepts rather than to diagnose. The other side is a genuine choice: blocking AI crawlers keeps your content out of model data, but it also keeps you out of citations in answer engines. You have to pick one, and both cost something.
Sources: Google: Overview of Google crawlers and fetchers Anthropic: does Anthropic crawl the web, and how can site owners block the crawler llms.txt proposal
Where this advice stops
This lesson is the big picture, not a debugging tool. If one specific page of your site will not index, the answer lives in that site's own Search Console reports, and no general explanation, including this page, replaces them.
From our own work
On this very site we allow sixteen AI crawlers by name in robots.txt, and llms.txt is rebuilt on every publish; the full list lives in wp-content/themes/rgb/inc/indexing.php and its output is visible at rgb.ir/robots.txt. That is a choice, not a default, and we know its cost: our content enters model data. We accepted it because being cited in answer engines is worth more to an agency.
Real follow-up questions
My page is indexed but gets no traffic. What does that mean?
It means the chain held to the third link and stopped at the fourth: Google has the page but knows a better answer for that query. This is not a technical error and never appears as one in any report; from here the work moves to search intent and to the quality of the answer.
What is the difference between robots.txt and a noindex tag?
robots.txt controls crawling; noindex controls being shown in results. If you block a path in robots.txt, Google never reads the noindex tag inside that page either, because it never fetches the page. To keep a page out of results, crawling must stay allowed so that noindex can be read.