robots.txt vs noindex: controlling crawling and indexing

Two of the most common technical SEO mistakes come from mixing up two different controls: robots.txt, which controls whether search engines may crawl a URL, and noindex, which controls whether a page may appear in search results.

robots.txt: crawling

The robots.txt file at the root of a site tells crawlers which paths they may request. A Disallow rule stops compliant crawlers from fetching those URLs.

What it does not do is keep a URL out of search results. Google's documentation is explicit: a page blocked by robots.txt can still be indexed if other pages link to it. It then appears without a description, because Google could not read the page.

Use robots.txt for:

• keeping crawlers out of areas that waste crawling, such as internal search results or endless filter combinations;
• reducing load on the server from unnecessary crawling.

noindex: indexing

A noindex rule, in a <meta name="robots" content="noindex"> tag or an X-Robots-Tag: noindex HTTP header, tells search engines not to show the page in results.

The catch: search engines have to crawl the page to see the noindex. If the same URL is also blocked in robots.txt, the noindex is never read, and the page can stay in the index.

Google also does not support noindex rules written inside robots.txt; that support ended in 2019.

Use noindex for:

• pages that visitors need but searchers do not, such as thank-you pages, internal search results, account pages or thin tag pages;
• staging sites (better still, protect them with a password).

Common mistakes

• Disallow plus noindex on the same URL. The page cannot be crawled, so the noindex is never seen. Remove the Disallow until the page has dropped out, then decide whether to block it.
• Blocking CSS and JavaScript. If Google cannot fetch the files a page needs, it cannot render the page properly.
• A staging Disallow left on the live site. One line in robots.txt can stop the whole site being crawled after launch.
• Using robots.txt to hide private content. robots.txt is public, and it is a request, not a lock. Use authentication for anything private.

Which to use

Check how Google sees any URL with the URL Inspection tool in Google Search Console, and see A technical SEO checklist for new sites for the launch-day checks.