Learning technical SEO, layer by layer

What layers is technical SEO made of?

Four layers, and each is a condition for the next. A page has to be reachable to be crawled, crawled to be rendered, rendered for its text to be indexed, and indexed before the question of speed means anything at all.

Here is a point that most checklists get backwards. Google's own order, according to the diagram in its own JavaScript documentation, is crawling, then rendering, then indexing. Rendering sits between the two, not after indexing. For a page whose text is built by JavaScript, that single swap decides whether the text enters the index at all.

The order you learn in, though, is not the machine's order, and it does not need to be. You learn from the bottom, because each layer is the condition for the one above it; but you notice problems from the top, because speed is the only layer you and your client can see. Someone who confuses those two spends six months on speed while the page was never indexed.

And one boundary that saves a lot of time: which of these layers moves rankings most is a different argument, and we wrote it separately. What technical SEO is and which part of it actually matters pulls that prioritisation out of real projects. This lesson does not prioritise; it gives a learning path and, at each layer, the check you run.

The crawl layer: how do I know Google reaches the page?

Crawling means Google's crawler finds the address, is allowed to fetch it, and gets an answer from the server. Three separate conditions, and each one breaks on its own.

Finding comes from links. A page with no internal link pointing at it, present only in the sitemap, is a marginal page as far as Google is concerned. The first check is always this: starting from the home page, how many clicks does it take to reach it?

Permission comes from robots.txt, and here is a rule almost nobody writes down. Google says only one group is valid for a particular crawler: the crawler picks the group whose user agent matches it most specifically, and ignores the other groups. The next sentence in the documentation is even blunter: specific groups and the global group are not combined.

Which means that if you wrote a separate group for one named bot, that bot sees none of the rules in the asterisk group. Read your own robots.txt once with that in mind; there is a fair chance that some rules you thought applied everywhere are written for only part of your crawlers.

Do not drop the second half of the same rule either, because you need it the moment you read a real file: several groups all written for one crawler are merged together. So in a file that declares the asterisk group three times, a crawler with no specific group of its own gets the rules of all three at once. The first half of the rule tells you which group to ignore, the second tells you which groups to read together, and someone who knows only the first half misreads a real file.

The third condition, the server's answer, is the simplest and eats the most time: a 200 for a page that should stay, a 301 for one that moved, and a 404 for one that genuinely is not there. A page whose content is gone but which still returns 200 is what gets called a soft error, and it is worse than either of the other two.

Why blocking in robots.txt is not the same as noindex

If only one thing from this lesson survives, let it be this section.

robots.txt says do not read. The noindex rule says you read it, now do not show it in results. The first is a crawl-layer control and the second is an index-layer control, and if you put both on one page, the second is never read.

Google states this with total clarity: for the noindex rule to be effective, the page must not be blocked by a robots.txt file and has to be otherwise accessible to the crawler; if it is blocked, the crawler will never see the rule and the page can still appear in results, for example when other pages link to it.

So a page you want out of the results has to stay open, so the crawler can come in and see the tag. That is precisely the opposite of what intuition says, which is why somebody makes this exact mistake every month.

In a different document, the one about duplicate URLs, Google comes back to the same thing and writes: do not use robots.txt for canonicalisation, because Google may still index URLs disallowed in robots.txt without their content. Two independent pages, one message: blocking is not removing.

The render layer: does Google see what you see?

Your browser runs the JavaScript and the page completes. Google runs it too, just not at the same moment.

Google's own documentation says the crawler queues pages both for crawling and for rendering, and adds that it is not obvious from outside whether a page is waiting for crawling or waiting for rendering. So between the moment Google takes your page's code and the moment it sees the text your script built, there is a gap whose length you do not set.

The check for this layer is the simplest of all and needs no tool. Take the raw code of the page, not what you see in your browser's developer tools, and search inside it for one sentence from the main body. If the sentence is there, the render layer is not your problem. If it is not, your text exists only for a browser.

Most tutorials lose the difference between those two: the code you see in the Elements tab of developer tools is the rendered code and it is always full. The thing that matters is the answer the server sent with no script running at all.

And if the sentence is missing, the answer is not always rewriting the whole site. For most content sites it is enough that the main text arrives in that first server response, while the rest of the page, the comments and the recommendations, gets built by script afterwards.

The index layer: the page exists, the result does not

Being indexed is not a right, it is Google's choice. A page that was crawled and rendered may still not enter the index, and the most common reason is not technical at all: the page holds nothing that is not already in the results.

The commonest technical reason, though, is a duplicate address. A product reachable through three category paths has three addresses; the www and non-www versions, the trailing-slash and no-slash versions, and every address built by a filter parameter add to the pile. Google picks one of them and sets the rest aside, and its pick is not necessarily yours.

Google lists the ways of stating your pick in order of strength: a redirect is a strong signal, a canonical link annotation is a strong signal, and inclusion in a sitemap is a weak signal. There are three more that never work and get tried anyway: robots.txt, the URL removal tool, and hope.

The word repeated in that list is signal, not instruction. A canonical tag does not force Google, which is exactly what makes people angry about it. If two addresses genuinely hold different content, putting a canonical on one of them does not make the problem go away.

And one practical warning that comes down to telling two problems apart: a duplicate address is one thing, and two separate articles competing for one phrase is another. For the second, a canonical is not the answer. If you are not sure which one you have, start with submitting a site to Google and the indexing problem, which opens that diagnosis from the ground up.

The speed layer is deliberately short here

Speed is the fourth layer and it gets two paragraphs here, no more. The reason is that site speed is a whole path of its own, and the right place to talk about it is the site speed path, not one section of a technical SEO lesson.

What you need from this lesson is one sentence: speed is the layer that starts to matter once the three below it are healthy, and not before. A page that is not indexed gets zero visits with a score of one hundred. The reverse holds too; a page whose first three layers are fine and which loads slowly has a problem its reader feels before Google does.

A multilingual site adds one more layer

If your site is monolingual, skip this section. If it is not, this is where most projects lose time.

When the same page exists at four addresses, all four have to be declared to each other, and each one has to list itself among the declarations as well. If the English version does not name itself in its own list, that relationship is incomplete. This is one of those things you check with one look at the page code and that almost nobody checks.

The second point, written down even less often, goes back to robots.txt. Path matching in that file is literal, which means writing one rule for a path does not cover the same path with a language prefix. A site whose languages are separated by an address prefix has to write every rule once per language, otherwise the language versions of that path are left with no rule at all.

And one thing that catches people out: the sitemap needs the same relationship. If your SEO plugin puts only the primary language's addresses in the sitemap, the other languages are found through internal links alone, and they become exactly the marginal pages described in the crawl layer.

Where do I start, and how do I check each layer?

The order below is the one that gets run on a project, and it is the same one to learn in. Each stage has a check that either answers or does not; if it does not, the next stage is meaningless.

First, the link path: how many clicks from the home page reach the page. Second, permission: it is not blocked in robots.txt, and the group that actually applies to it is the group you thought applied. Third, the server's answer: the status code is the one it should be. Fourth, rendering: one sentence from the body is findable in the raw page code. Fifth, indexing: the page is in the index, and this is the one thing only Search Console can answer, not another tool and not a language model. Sixth, speed.

One warning about the checking itself, which comes from doing it and is in no tutorial: when you search the page code for the robots tag, watch the quote style. WordPress and SEO plugins print that attribute sometimes with single quotes and sometimes with double, and a search that knows only one form will happily report that the tag does not exist. You can see this on this very site.

And if you would rather not do this yourself, this same order is where we start in SEO work; not because it is elegant, but because working on the wrong layer is the most expensive mistake in this field.