Information Retrieval: The Science Behind Search Engines

How search engines work in my last blog article is something I explained. But how exactly does finding specific information within the countless amounts of data in a search engine work, so that relevant search results can be delivered to the user in a flash?

This is exactly where “information retrieval (the retrieval of information)” comes into play. It is, so to speak, the science behind search engines.

In this article, I'll show you what exactly this is and which models exist.

What is information retrieval and why is it relevant for SEO?

Information retrieval is a discipline within computer science, and its task is finding specific information from large, unfiltered amounts of data. This is exactly the task that a search engine has for the areas of crawler, indexer, and searcher.

In response to a search query, the searcher has to deliver the best possible search results for the user. To do this, it draws on the processed data from the index. The indexer has to process and sort the raw data from the crawler's local store very well. There are various models and measures for how such data is processed. But let's start first by addressing the problem of the unstructuredness of data:

Problems of a search engine

It is impossible for a search engine to deliver the correct or, respectively, the results relevant to the user from a large, unsorted dataset within a few milliseconds. The faster a search engine delivers its results, the more satisfied its users are – but of course only if the delivered results are relevant and fitting to the search query. In order for the search engine to quickly retrieve all stored data and display relevant results, this data has to be processed and sorted accordingly.

Another problem for search engines is imprecise search queries on the part of the users themselves. Imprecise queries can also cause the user to receive insufficient information. The system of Query Modification therefore modifies the search query automatically in order to display relevant search results. Among other things, a stop-word elimination is used here, so that, for example, the word “and” is ignored in a search query.

Further query modifications:

With the help of the thesaurus synonyms and related terms for the search query are derived.
Through multi-word group identification groupings of words are recognized as such.
With base- and stem-form reduction words are reduced to their word stems. Inflected forms of a term (e.g., Hut and Hutes) would otherwise not be displayed correctly.

Recall & Precision

The effectiveness of an information retrieval system is calculated using the two factors Recall (hit rate) and Precision (accuracy), which are shown as quotients (see graphic below).

Recall describes the completeness (the coverage) of the search results. It measures how many relevant documents are displayed in relation to all relevant documents stored in the dataset.
Precision describes the accuracy of the search results, i.e., the number of relevant documents found in relation to the total search results received for a query.

So the two quotients calculate the completeness and accuracy of the search results and can be represented in an axis diagram. Both values lie between 0 and 1. A 1 would be the perfect result. In that case, all relevant documents for a query would be displayed, and at the same time there would be no irrelevant document among them. In practice, however, this result is not achievable. Whoever increases the accuracy (precision) of the relevant search results thereby reduces their completeness (recall).

Example — Recall (hit rate) and Precision (accuracy)

Did you know?

Incidentally, the topic of information retrieval is older than the internet. As early as 1945, the US scientist Vannevar Bush described in his essay “As We May Think” how people could make their knowledge more accessible. In the 1950s, Hans Peter Luhn developed methods such as automatic indexing and full-text processing – foundations that are still built into search engines today.

Further challenges in delivering search results are shown below:

Relevance, pertinence, and usefulness

A search result can be relevant, but it does not have to be pertinent. Pertinence refers to the user's subjective perspective. If a user already knows the text of the relevant search result, for example, then it is not pertinent for them. A search result is only useful to the user once they gain new knowledge from it that they can use in practice.

So Google and the like have the task of delivering relevant search results that are useful to the user and also unknown (pertinent) to them, and that show them something new.

With the help of so-called retrieval models, search engines try to display the best possible search results for their users' search queries.

Retrieval models and ranking factors

In information retrieval, there are various models that are meant to help deliver suitable search results from a (large, unsorted) dataset. These models do not exclude one another; instead, they can be combined. Here is a selection of common models:

The Boolean model (set-theoretic model): Here, relationships are determined through set operations (AND, OR, and NOT).
The vector space model and the probabilistic model are models based on text statistics (WDF*IDF).
The link-topological models include, among others, the famous PageRank. Here, the links between documents are taken into account.

These models show which ranking factors a modern search engine works with. Of course, the exact weighting and composition of the result lists through the ranking factors is the trade secret of every search engine. Nevertheless, there are a few well-known and important ranking factors. Every search engine works differently here – Google states that it has over 200 ranking factors. Particularly important factors that a search engine takes into account in its rankings are text, links, locality, freshness, personalization, brand, user, trust, and technology.

Text: Which words from the search query appear in the document? Where and in what context do they appear in the document? Synonyms, images, headings, anchor texts, and much more can be taken into account here.
Links (also referred to as the popularity of documents): How heavily is a page linked, and from which documents does it receive these links? Google's PageRank algorithm measures the popularity of documents based on links.
Locality: Here, the searcher's location is taken into account. When searching for a dentist, for example, it is of little help to a user from Düsseldorf to be shown a list of search results with dentists from Berlin. The location of the document (language, company headquarters, etc.) is preferentially delivered for a search query with a local reference.
Freshness: Depending on the searcher's search intent, displaying especially up-to-date documents and information may be desired (e.g., for news, weather reports, sports results, etc.). Since older information is usually linked more heavily but may not have the desired freshness, the ranking factor “freshness” can favor documents that have fewer links but more up-to-date information.
Personalization: This is about the user's personal preferences. Search engines adapt the search results, for example, to which online shops the user has clicked on especially often in the past, etc.
Brand: How well known is the website among users, and how high is the brand traffic? A website that is known outside of search engines and is regarded by users as a reliable source can also rank better in the search results due to this brand effect.
User: An eternal point of contention is user data as a ranking factor. Google denies using such data, but tests increasingly show that user data such as click-through rates in the SERPs and other user behavior do flow into the rankings. Bing admits to measuring user data such as the bounce rate and time on page as signals for the rankings – whether Google also uses this, however, is not confirmed.
Trust: Can the website be trusted, so that it is shown to the user in the TOP 10? Google's E-E-A-T model is a further step so that the search engine, especially for particularly sensitive topics such as health and finance (YMYL), displays only pages whose content is correct and poses no danger to the user. EEAT is not a direct ranking factor in itself, but rather a concept used by the search engine to better identify trustworthy pages.
Technology: Search engines have to understand a website. Technical hurdles such as faulty links, excessively long loading times, faulty code, and many more indicate the poor quality of a website and can make it harder for the search engine to understand it. Only a technically clean website can be read correctly by the crawlers. Google itself states that during optimization, the overall quality of the website must be considered, not just individual documents.

Sources:

That was a brief overview of how a search engine actually works. Anyone who optimizes their website should always keep in mind how a search engine functions.

-Fabian

Need a GEO check?

Get in touch for more visibility – in search engines and AI answers.

More visibility and leads via Google and AI chats
Sustainable SEO & GEO strategies
Personal support

Free initial analysis →