Noindex – Definition & Explanation
The “noindex” entry in a website’s source code asks search engines not to include the page in question in their index. This lets you prevent URLs from being indexed. That is important in search engine optimization (SEO), for example, in order to avoid duplicate content.
When Does the Directive Make Sense?
There are several use cases for the noindex directive:
- A web page contains duplicate content. noindex can be one way to keep it out of the index – but for duplicate content the canonical tag is often the more suitable means.
- A page is being revised. With noindex, webmasters can work on new pages undisturbed before they are indexed. The directive has to be removed again in good time, otherwise the page stays permanently undiscoverable.
- Copyrights are to be protected.
How Noindex Is Applied
There are two options for using it.
1. Through a meta tag in the head of the source code. If the general Googlebot is to be kept from indexing:
<meta name="googlebot" content="noindex">
It works the same way for other crawlers. To address all search engines, webmasters use
< meta name="robots" content="noindex" >
Google uses various bots; website operators can find an overview of them in the Google Help.
2. Via the HTTP response header (X-Robots-Tag). This, too, can dissuade various crawlers from indexing. The result is the same in both cases; what matters is which option is more practical.
An alternative to the meta tag is the X-Robots-Tag: via the HTTP header, noindex (and, for example, nofollow) can also be set for non-HTML files such as PDFs, images, or videos – something the meta tag in the <head> cannot do.
The noindex directive is most reliably checked via the URL inspection in the Google Search Console (or by checking the HTTP header/source code); the site: query only provides a rough indication.
The URL inspection in the Google Search Console lets you reliably verify whether a page is indexed and whether the noindex is correctly recognized – more precisely than the site: query.
Noindex,follow VS. Noindex,nofollow
The Noindex directive can be supplemented with “follow” or “nofollow”:
<meta name="robots" content="noindex,follow">
signals to the crawler not to index the page but to follow the links on it. This is useful, for example, on internal search results pages or paginated category pages whose products are meant to be indexed.
Note on the long-term effect: Over time, Google effectively treats a page that is permanently set to “noindex” like “noindex,nofollow” – in the long run, the links on the page are no longer followed. For paginated pages or internal search results, clean internal linking or canonical tags are therefore often the more sustainable solution.
<meta name="robots" content="noindex,nofollow">
is rarely used and instructs crawlers not to follow the links on the page. Important: “nofollow” does not prevent crawling – the attribute only instructs the crawler not to follow the links; the page itself is still crawled. You prevent a page from being crawled exclusively via “Disallow” in the robots.txt.
Noindex or robots.txt: When Does Which Fit?
Besides noindex, crawlers can also be prevented from crawling via robots.txt files. However, the page can still appear in the search results if other pages link to it. Google advises using robots.txt only to:
- regulate crawling traffic
- prevent image, video, or audio files from appearing in the SERPs
- block unimportant resource files
A common conflict is the combination of noindex and Disallow: if a page is blocked for the crawler via Disallow in the robots.txt, the crawler cannot read the noindex on the page at all. Through backlinks, the page can then still end up in the index (GSC status “Indexed, though blocked by robots.txt”). noindex only takes effect if the page remains crawlable.
In addition: noindex in the robots.txt has no longer been supported since September 1, 2019. Google switched off the unofficial noindex directive in the robots.txt on that date. Since then, noindex belongs in the meta tag or the X-Robots-Tag, not in the robots.txt.
Quellen
SEO & GEO check
Want more visibility – in search engines and AI answers?
We analyse your potential and show concrete next steps. Free and non-binding.