Free, no sign-up

robots.txt and Sitemap Check

Your domain robots.txt is read and tabulated, the declared sitemaps are opened and their URLs counted, and the recurring mistakes that hide a site from Google are flagged.

  • No sign-up
  • Sitemap URL counts
  • Common mistakes flagged

Run the tool right here

See what Googlebot is allowed to crawl and whether your sitemap is healthy.

  • What it costs Free
  • Data source Our own server
  • Access No sign-up

This tool runs on our own server and we buy no data for it, so it is free and stays free. It only carries a daily cap so a bot cannot swallow the whole capacity. The daily cap is counted per visitor. If you need more, a free account raises your limit.

The short answer

The robots.txt file tells crawlers which parts of the site not to crawl, and the sitemap tells them which URLs exist and when they changed. If a page of yours is not getting indexed, these two files are the first place to look, before anything else.

The worst thing that happens to a site is one line: Disallow: / . Every staging environment has that line and rightly so, but when the site moves from staging to the live server it usually travels with it and removes the whole site from Google. No warning appears anywhere; traffic just drifts to zero over a few weeks and nobody knows why. Every time you migrate a site, check this on day one.

The mistake we keep seeing on Iranian WordPress sites is blocking wp-content. The logic was once right and is not any more: to work out whether your page is mobile friendly, Google has to see its CSS and JavaScript, and those all live inside wp-content. Block that folder and Google sees the page with no styling and judges a shapeless page.

On sitemaps, the number worth reading is the gap between the URL count inside the sitemap and the indexed page count in Search Console. If the sitemap has a thousand URLs and Google has indexed two hundred, the sitemap is not the problem; Google has seen eight hundred pages and decided they were not worth indexing. The right question then is about the quality of those pages, not about the sitemap file.

Where the data comes from

Our server fetches the file directly from your domain; no third-party service is involved and there is no cost. If robots.txt declares no sitemap, the conventional paths (sitemap.xml, sitemap_index.xml, wp-sitemap.xml) are tried too. Something we keep seeing: many Iranian WordPress sites still block wp-content, which makes Google render the page without CSS and judge it not mobile friendly.

When it helps

The first thing to check when a page will not get indexed. Also after any site migration or host change, because a staging robots.txt usually carries Disallow: / and wipes the whole site out of Google.

Where it does not help

This tool says what robots.txt permits, not what Google actually did. A page can be fully allowed in robots.txt and still not be indexed, because Google did not think it worth it or its canonical points elsewhere. The reverse also happens: a page blocked in robots.txt but linked from elsewhere sometimes appears in results with an empty title, because robots removes permission to crawl, not permission to index. For do not index, the right instrument is the noindex tag, not robots.txt. To see what Google actually did with a URL, the only correct source is that site own Search Console.

One thing we learned using it

If robots.txt does not declare a sitemap, this tool also tries the conventional paths: sitemap.xml, sitemap_index.xml and wp-sitemap.xml. We added that deliberately, because in practice we have repeatedly seen a sitemap that exists and is healthy but is declared nowhere, with the owner believing there is none. Declaring the sitemap in robots.txt is one line, and that one line is what tells a crawler arriving at your site for the first time where to start.

What it costs

This tool runs on our own server and we buy no data for it, so it is free and stays free. It only carries a daily cap so a bot cannot swallow the whole capacity.

The daily cap is counted per visitor. If you need more, a free account raises your limit.

Questions about this tool

Is it a problem if I have no robots.txt?

Not necessarily. No file means everything is allowed, which for most small sites is exactly what they want. The file becomes necessary when you want to keep sections such as internal search pages or shop filters out of crawling.

Can Disallow remove a page from Google?

Not reliably. If the page is linked from elsewhere it may still sit in results with an empty title. For real removal use the noindex tag, and more importantly do not block that same page in robots.txt, because Google has to be able to crawl the page to see the noindex.

How often should a sitemap be updated?

Automatically, with every new piece of content published. Almost every WordPress SEO plugin does this. A hand-built sitemap that has not been updated in months is worse than none, because it sends the crawler to URLs that no longer exist.