Crawler (Web Crawler): Definition & Explanation

A crawler is a computer program that automatically searches the internet for content and analyzes it. Search engines like Google use crawlers to build an index that users can search.

The term crawler goes back to one of the first popular web search engines, WebCrawler (1994). It was not, however, the very first search engine: earlier predecessors were Archie (1990, searching FTP file listings) and JumpStation (December 1993, the first crawler-based full-text search engine for the web). Alternative terms include “bot” or “spider,” for example.

Crawlers are also used in web analytics and data mining. Users have various options for controlling or blocking them. Knowing about these options pays off, for example, in search engine optimization (SEO).

How does a web crawler work?

Crawlers usually move from one web page to the next via hyperlinks. As they go, they gather information and then sort it. This creates an index that software can access in order to retrieve and evaluate information. Only with crawlers can a search engine SERPs provide.

Crawlers work automatically. How they proceed varies depending on the “client.” One especially well-known example is Google's crawler, the so-called Googlebot. It exists in several versions. Besides the Googlebot for computers, there is Googlebot Mobile, but also Googlebot News and Googlebot Images.

It exists in several versions. Since the completion of mobile-first indexing (2023/2024), Google crawls primarily with Googlebot Smartphone; Googlebot Desktop is secondary. The official crawlers include, among others, Googlebot Smartphone, Googlebot Desktop, Googlebot Image, Googlebot News, and Googlebot Video.

Apart from crawlers that work for search engines, there are web crawlers that

collect email addresses or postal addresses for marketing purposes,
search for products and their associated data for price comparison portals,
collect statistical data or news.

This crawling is often unwanted by website operators and legally problematic.

Crawlers in general should be distinguished from scrapers. The latter extract content from web pages. The range spans from harmless scraping — for example, to publish transit schedules on websites or for analytics purposes — all the way to deliberate copyright infringement in order to obtain content quickly and easily.

Today, crawling means more than just following hyperlinks. Googlebot renders JavaScript in a second step (rendering) in order to capture content loaded via JavaScript. If CSS or JS files are blocked via robots.txt, this can degrade rendering and thus indexing — these resources should remain crawlable.

The importance of crawlers for SEO

Crawlers have an important function for search engine optimization (SEO). Regular crawling is necessary so that all relevant parts of a website are indexed and appear in the search results.

Crawlers only have a limited amount of time available for each page. This is also referred to as the crawl budget.

Google defines the crawl budget as the interplay between crawl capacity (crawl rate limit — how much the server can handle) and crawl demand (how sought-after and up-to-date the content is). It is relevant above all for very large websites (from around 10,000 URLs); small sites generally do not need to worry about it.

So that crawlers make better use of this budget or that it is increased, webmasters should

optimize page structure and navigation,
work to attract many incoming links and visitors, and
keep file sizes as small as possible.

In addition, it is advisable to submit an Google Search Console a XML sitemap for your own site in the Google Search Console, or to link the sitemap in a robots.txt file (via the Sitemap: directive). In the Search Console you can also check whether a crawler has indexed all areas of the website.

Blocking crawlers

To block crawlers from certain content, there are several options:

With a robots.txt file, you can tell crawlers which areas or content of websites they are allowed to crawl. However, the pages in question can still appear in Google Search.
If you want to completely prevent indexing and the appearance of pages in the SERPs, it is best to add the meta tag noindex to a page's HTML code, or alternatively to return the X-Robots-Tag noindex in the server's HTTP response header.
Finally, crawlers can be blocked using password-protected server directories.

Crawlers that harvest email addresses generally do not comply with such rules. Here you have the option of making email addresses unrecognizable to the crawler. To do this, many website operators use a notation such as company(at)domain(dot)com.

AI Bots and Robots.txt

Since 2023/2024, training and AI search bots such as GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended, and PerplexityBot have been a central topic. They too can be controlled specifically via robots.txt by setting separate Disallow rules per user agent — separately from the classic search engine crawlers.

Here is an example of a robots.txt that blocks AI bots, either fully or partially:

Example: a robots.txt that blocks AI bots

# =================================== # STANDARD SEARCH ENGINES (allowed for normal search results) # =================================== User-agent: Googlebot User-agent: Bingbot Allow: / # =================================== # AI TRAINING AND SEARCH BOTS (completely blocked) # =================================== # OpenAI (uses data for training ChatGPT) User-agent: GPTBot Disallow: / # OpenAI search bot (for web search within ChatGPT) User-agent: OAI-SearchBot Disallow: / # Anthropic (uses data for training Claude) User-agent: ClaudeBot Disallow: / # Google AI (uses data for training Gemini/Vertex AI) # Note: does NOT block the normal Googlebot for search! User-agent: Google-Extended Disallow: / # Perplexity AI (AI search engine) User-agent: PerplexityBot Disallow: / # Apple AI (uses data for Apple Intelligence) User-agent: Applebot-Extended Disallow: / # Common Crawl (large open web database used by many AI models) User-agent: CCBot Disallow: / # =================================== # EXAMPLE OF PARTIAL ACCESS (hybrid approach) # =================================== # If a bot (e.g. from Meta) should read the articles # but not scan any internal data or images: User-agent: Meta-ExternalAgent Disallow: /admin/ Disallow: /uploads/ Disallow: /private/

The difference between robots.txt and noindex

robots.txt and noindex have different functions: robots.txt controls crawling (access), noindex controls indexing (appearing in the search results). A page blocked via robots.txt cannot be crawled by Google at all, and therefore Google never reads a noindex set there.

If you want to reliably keep a page out of the index, you should therefore leave it crawlable and mark it with noindex, instead of blocking it via robots.txt.

Spoofing the User-Agent

The user agent alone can be spoofed — spam bots like to pose as Googlebot. Genuine Googlebot accesses can be verified via a reverse DNS lookup and the IP ranges published by Google, before you block or exempt anything based on the alleged crawler.

Quellen