A robots.txt file does not tell Google what to index. It tells Google what not to crawl. Those are different instructions, and that difference is behind most of the mistakes made with this file.
We checked 500 Persian domains. 463 serve a real robots.txt, 25 do not answer with a 200, and nine return an HTML page instead of text, meaning the crawler reads that site's 404 template where its directives should be.
What is actually inside those files
| Across 463 real files | Count |
|---|---|
| Carry a Sitemap line | 397 (86%) |
| No Disallow rule for everyone at all | 81 |
| Block at least one AI crawler | 18 (4%) |
| Use crawl-delay | 5 |
| Block the whole site for everyone | 1 |
| Carry Cloudflare's standard block of text | 14 |
The median file has nine directive lines and the largest has 1,518. The median number of Disallow rules under the wildcard is four; the largest is 139.
Three findings that contradict the common picture
Almost nobody is blocking AI crawlers
Eighteen of 463. If you have heard that Iranian sites are shutting the door on AI, they are not. And those eighteen have essentially copied one list: fifteen block GPTBot, ClaudeBot and Bytespider; fourteen block CCBot, Amazonbot and Google-Extended; thirteen block Applebot-Extended. One block, passed around.
Fourteen sites did not write their own file, and one blocks itself entirely
Fourteen robots.txt files carry Cloudflare's content signals policy text, the comment block that appears when you turn on an option. The site owner does not necessarily know it is there. If you sit behind Cloudflare, open your own file once and see what has been written on your behalf.
We first thought eighteen did. That was wrong: our pattern crossed block boundaries and counted a Disallow: / written for one specific bot as if it applied to everyone. Parsing the files block by block brought the real number down to one.
What this file does not do
A page blocked in robots.txt can still appear in Google's results, because Google knows it from external links and has simply never read its content. The result is that description free listing everyone has seen.
This is where getting a site indexed gets confused with getting it crawled. To keep a page out of results the right tool is a noindex tag, and that tag is only read if the crawler is allowed to fetch the page. So blocking the path in robots.txt is precisely what stops noindex from being seen. The two do not work together.
What to block and what to leave alone
Block: internal search results, login and registration pages, and parameter URLs that produce duplicate versions of a page. None of it deserves crawling and all of it consumes crawl budget.
Do not block: CSS and JavaScript files. Google renders the page, and if it cannot see your stylesheet it sees a broken site. That mistake still shows up in older themes.
And do not write crawl-delay. Five sites in our sample did; Google has never supported the directive and ignores it. If crawling is straining your server, the problem is whatever the site is sitting on, not a line of text.
Our own file, and why it looks the way it does
This site's robots.txt runs to 105 lines in three parts. First, the worthless paths are closed: internal search in all four languages, login and registration in all four, and the return parameter.
Then a block headed AI crawlers welcome, explicitly allowing GPTBot, ClaudeBot, PerplexityBot and the rest. That is a decision rather than a default: we would rather our text be quoted in those answers.
Third, the private proposal path is closed to everyone but opened to link preview bots, because without that exception the link we send a client on WhatsApp arrives with no title and no image. We learned that one the hard way, once.
What this count cannot tell you
We counted the presence of directives, not their correctness. A site that has mistakenly blocked something important counts in our figures as a site with a healthy robots.txt. The files were also read on one day from one point, and some sites serve a different file depending on IP or user agent.
Questions people actually ask
What happens if I have no robots.txt?
Nothing bad. No file means everything is allowed. Eighty one sites in our sample have a file with no wildcard rules at all, which amounts to the same thing.
Do I need the Sitemap line?
Not required, but useful, and 397 of 463 carry it. Search Console is a more reliable way to declare a sitemap; the robots.txt line is for the other engines.
Should I block AI crawlers?
That is a commercial decision, not a technical one. If your revenue depends on direct visits, blocking makes sense. If you want to be quoted in answers, allowing does. We chose the second and said so explicitly in the file.
Does blocking a path remove it from Google?
No, and it sometimes does the opposite. If a page is indexed now and you want it gone, add noindex first, wait for the removal, then block the path if you still need to. The order matters, and in technical SEO work getting it backwards is the single most common reason a page stays in the results.
If you are unsure what your file actually says, send us its address. We will read it block by block and tell you which rules are really in force.
Measured 3 September 2026 across 500 Persian domains from Google results. Each file was parsed block by block, not pattern matched across the whole text.
Comments & Questions
Have a question about this article? Ask, we'll answer.
No comments yet; be the first.