robots.txt & Sitemap Validator

One line can remove a site from search

Disallow: / under a wildcard user agent tells every crawler to read nothing. It is a single line, it is correct syntax, and it is the most common reason a new site never appears in search results. It usually arrives by accident, left over from a staging server that was deliberately hidden and then copied to production with everything else.

The reverse mistake matters too. robots.txt controls crawling, not indexing. A page that is blocked here can still appear in results if other sites link to it, listed without a description because the crawler was never allowed to read it. If a page must stay out of search, the noindex meta tag is the instruction that does that, and the crawler has to be allowed in to see it.

Do not block your stylesheets

An older habit was to block directories full of theme files to keep crawlers focused on content. Google now renders pages the way a browser does, and judges mobile usability on the result. If the stylesheets and scripts are blocked, it renders a broken page and grades the site on that. Any rule that blocks CSS or JavaScript is almost certainly costing more than it saves.

What a sitemap is actually for

It does not improve rankings. It tells crawlers what exists, which matters most for pages that nothing links to yet, large sites where crawling takes a long time, and new sites with no inbound links at all. Keep it accurate: a sitemap listing pages that return 404, or pages blocked in robots.txt, sends contradictory signals and wastes the crawl budget you were trying to direct.

One file holds up to 50,000 URLs and 50 megabytes uncompressed. Past that, split it and list the parts in a sitemap index. Every address must be on the same domain and should match the canonical form exactly, so if the site runs on https with no www, every entry in the sitemap should look the same way.