How Do Search Engines Rank Web Pages? | How It Works

What is the short answer?

A web search engine does most of its work before the student types anything. Programs called crawlers fetch pages by following links and lists of addresses called sitemaps. The engine stores what each page says in an index, which lists every word with the pages that contain it.

When a query arrives, the engine works out what the words mean, pulls the matching pages from its index and puts them in order. Google says relevance is judged by hundreds of factors.

Definition: Ranking is the ordering of the pages that match a query. Software weighs many signals about each page, such as how well it matches the words, which pages link to it and how fresh it is.

How does a search engine find pages in the first place?

There is no central register of the web. Google says its crawler, Googlebot, discovers new addresses through links on pages it already knows and through sitemaps that sites submit. Microsoft's crawler is Bingbot.

A site can post a file called robots.txt asking crawlers to stay out of some paths. Martijn Koster first set out this Robots Exclusion Protocol in 1994. The IETF published it as a standard, RFC 9309, in September 2022.

The standard says crawlers are requested to honour the rules, which are not a form of access authorization. If robots.txt returns an error such as 404, the crawler may fetch anything; if the server fails, it must assume everything is blocked. A cached copy should not be used for more than 24 hours.

What does a search index actually store?

The engine reads each fetched page's text and key parts, such as the title. Google compares its index to the index at the back of a book, with an entry for every word on every page it indexes. It says the index is well over 100,000,000 gigabytes.

This structure is an inverted index. A normal file lists the words on each page; an inverted index flips that, listing for each word the pages where it appears. Google's 1998 prototype recorded each word's occurrences in a page, with position, font and capitalisation, in a hit list. A two-word query then means comparing two lists, not rereading every page.

Engines also handle copies. Google groups similar pages and picks one canonical page that may be shown, and says indexing is not guaranteed.

How does the engine work out what the searcher meant?

Google's How Search Works pages name jobs such as correcting spelling, finding synonyms so that pages count without the exact words typed, and identifying the intent behind the query. Google says the system behind these jobs took over five years to build and improves results in over 30% of searches across languages.

Intent can change the best answer. For "earthquake", pages on preparation usually serve, but after a recent earthquake, Google's "query deserves freshness" systems may lift news.

Machine-learned models now help. A Google post of February 2022 dates three: RankBrain (2015), its first deep learning system in Search; neural matching (2018), used to retrieve relevant documents; and BERT (2019), which Google says plays a role in almost every English query.

How does the engine decide which page comes first?

Relevance starts with matching terms. Google's most basic signal is a page containing the query's keywords, especially in headings or body text. Microsoft lists relevance first for Bing, matching words on the page and in links pointing to it.

Links give a second view. Treat each page as a dot and each link as an arrow, and the web becomes a link graph. In 1998, Sergey Brin and Lawrence Page of Stanford University described PageRank.

Each page passes its score to the pages it links to, split evenly across its outgoing links and scaled by a damping factor they usually set to 0.85. They pictured a "random surfer" who keeps clicking links but eventually gets bored and starts again on a random page; the chance of being on a page is its PageRank.

Google still calls PageRank a core ranking system, but says it has evolved a lot and works among many systems. Context counts too. In Google's example, "bicycle repair shops" gives different results in Paris and in Hong Kong.

Bing weighs the searcher's country and city, the page's language, freshness and load time. Microsoft says Bing learns from user interactions and from examples labelled by human judges.

How do search engines deal with spam and test quality?

Where ranking brings visitors, some people fake the signals. Google's spam policies name keyword stuffing, filling a page with keywords out of context; cloaking, showing search engines different content from users; and link spam, making or buying links to manipulate rankings.

Google detects these with automated systems, including SpamBrain, and with human review. Sites that break the rules may rank lower or not appear at all.

To test ranking changes, Google works with external Search Quality Raters. In side-by-side tests, they compare two sets of results, one with the change, and say which they prefer. Google states that these ratings do not directly impact ranking. For 2023, it reports 124,942 side-by-side experiments and 4,781 launched changes.

Do search engines in other countries work the same way?

The same three stages run everywhere, but each engine publishes its own rules and names its own crawler, so robots.txt can address one engine without affecting others.

• Baidu, China. Its company page says it was founded on 1 January 2000 and that its founder, Li Yanhong (Robin Li), holds a patent for "hyperlink analysis" technology.
• Yandex, Russia. Its help pages describe a Clean-param directive, which the Yandex robot uses to ignore address parts, such as tracking tags, and avoid reloading duplicates.
• Naver, South Korea. Its integrated search, from 2000, arranges results by the format and source of content, such as blogs, community "cafes" and images. Its crawler, Yeti, follows the same 404 and server-error rules as RFC 9309.
• Seznam, Czech Republic. SeznamBot fetches pages for its search engine, while a fast crawler, FreshBot, indexes fresh content, mostly from news sites.

Why do courts and regulators care about ranking?

In its Google Shopping decision, the European Commission recorded that the ten highest general results on Google's first page together receive about 95% of clicks on general results, and the first result on page 2 only about 1%.

• European Union, Google Shopping. On 27 June 2017, the Commission fined Google €2.42 billion for abusing its dominance as a search engine: it promoted its own comparison shopping service and demoted rivals, the best of which appeared on average on page four. The General Court largely upheld the decision and kept the whole fine on 10 November 2021; the Court of Justice dismissed Google's appeal on 10 September 2024.
• European Union, Digital Markets Act. On 6 September 2023, the Commission designated six gatekeepers, including Alphabet for Google Search; in February 2024 it decided not to designate Bing. Gatekeepers must not favour their own services in ranking. On 23 July 2026, it fined Google €460 million for favouring its own services, including shopping, hotels, transport and sports results. Google may appeal.
• United States v. Google. The Justice Department and a group of states sued in October 2020. In August 2024 the court found that "Google is a monopolist", breaching the Sherman Act. Remedies ordered in September 2025 bar exclusive distribution deals and make Google share some index and user-interaction data with certain competitors; the final judgment followed on 5 December 2025. The case is on appeal and cross-appeal; briefs were filed in October 2026.

How does this work with real numbers?

Both examples use tiny, illustrative pages and links, checked with a short Python program.

How does an inverted index answer a two-word query?

Page 1 says "Volcanoes erupt when magma rises". Page 2 says "Magma cools into new rock". Page 3 says "Volcanoes build new islands".

Table: Part of the illustrative inverted index

The full index holds 11 different words. For "volcanoes magma", the engine fetches two lists: pages 1 and 3, and pages 1 and 2. They overlap only at page 1, the one page with both words.

Counting matching words scores page 1 at 2 and pages 2 and 3 at 1 each, a tie that other signals, such as links, must break.

How does PageRank settle on four pages?

A links to B and C; B links to C; C links to A; D links to C. Each page starts at 0.25. Each round, a page's new score is 0.15 ÷ 4 = 0.0375, plus 0.85 times the shares it receives; each page splits its score evenly over its links.

In round 1, C receives half of A's 0.25 and all of B's and D's: 0.0375 + 0.85 × (0.125 + 0.25 + 0.25) ≈ 0.57.

Table: Illustrative PageRank scores with a damping factor of 0.85

After 17 rounds, no score moves by more than 0.0001, and the scores add up to 1. C, with three incoming links, leads. A has one, but it comes from C, so A is second. D has none and keeps only the 0.0375 base.

The 1998 paper writes the formula without dividing by the number of pages. That multiplies every score by 4 but leaves the order unchanged.

Where can the student go deeper in World 101?

This page follows one query from the crawler to the results page. The World 101 subject Algorithms and Computation covers the wider field: how step-by-step procedures are designed, how their cost is measured and how they come to make decisions that once needed people.

Three related subjects go further. Networks and Graphs studies dots and the links between them, the structure that PageRank reads. Recommendation Algorithms looks at how feeds and shops choose what to show each person, a close relative of search ranking. Markets, Competition and Firms explains dominance and competition, the ideas behind the cases in Brussels and Washington.

Sources

• IETF, RFC 9309, Robots Exclusion Protocol, September 2022 (first defined by Martijn Koster, 1994).
• S. Brin and L. Page, "The Anatomy of a Large-Scale Hypertextual Web Search Engine", Computer Networks 30, 1998 (Stanford InfoLab text, archived copy; Google Research listing).
• Google Search Central: how Google Search works, updated 18 December 2025; ranking systems guide, updated 10 December 2025; spam policies, updated 28 August 2026.
• Google, How Search Works pages (testing figures for 2023), read 9 November 2026.
• P. Nayak, "How AI powers great search results", Google blog, 3 February 2022 (RankBrain 2015, neural matching 2018, BERT 2019).
• Microsoft Support, "How Bing delivers search results", March 2025.
• Baidu company page (founded 1 January 2000); Yandex Webmaster Help on Clean-param; NAVER About (2000) and Search Advisor robots.txt guide; Seznam.cz SeznamBot help and About Seznam page; all read 9 November 2026.
• European Commission, IP/17/1784, 27 June 2017; Court of Justice, C-48/22 P, 10 September 2024, on the General Court's judgment of 10 November 2021.
• European Commission, DMA gatekeeper designations, 6 September 2023; Bing decision, 13 February 2024; Google DMA fines, 23 July 2026.
• US Department of Justice, Google search remedies release, 2 September 2025 (ruling of August 2024); case page, updated 2 November 2026 (complaint October 2020, final judgment 5 December 2025, appeal briefs October 2026).

Common errors and misconceptions

• Misconception: A search engine searches the live web when a query is typed. Correct: It searches its own index, built earlier; a page it never crawled or chose not to index cannot appear.
• Misconception: A robots.txt file keeps a page private. Correct: RFC 9309 says its rules are not access authorization; crawlers are asked to honour them, and anyone can still open the page.
• Misconception: The page with the most links always comes first. Correct: PageRank also weighs where links come from, and Google calls it one system among many.
• Misconception: Human quality raters push individual pages up or down. Correct: Google says rater scores do not directly impact ranking; they test whether a proposed change improves results.
• Misconception: Everyone who types the same words sees the same results. Correct: Results depend on context such as location, language and device.

Key takeaways

• A search engine works in three stages: crawling pages through links and sitemaps, indexing what they say, and ranking matches when a query arrives.
• Robots.txt, an IETF standard since RFC 9309 in 2022, asks crawlers to skip pages; it is a request, not a lock.
• An inverted index lists every word with the pages that contain it, so answering a query means comparing short lists.
• PageRank scores a page by the links pointing to it, weighted by each linker's own score; Google calls it one system among many.
• Machine-learned systems such as RankBrain, neural matching and BERT match queries to pages by meaning, not only exact words.
• The European Commission has fined Google for favouring its own services in top places, which draw most clicks, and a United States court found it a search monopolist.

Test yourself

Why might a page that is online never appear in search results?

Crawlers may never find it, robots.txt may ask them to skip it, or the engine may crawl it but not index it.

In the illustrative index, how are pages with both "volcanoes" and "magma" found?

The engine compares the two word lists, pages 1 and 3 and pages 1 and 2, and keeps the overlap: page 1.

Why does page A come second in the PageRank example?

Its one link comes from C, the highest-scoring page, which passes its whole weighted share to A.

What does a search engine do to a query before ranking pages?

It corrects spelling, finds synonyms and works out the intent, such as whether fresh results are wanted.

What did the European Commission find in its Google Shopping decision of 2017?

That Google abused its dominance as a search engine by promoting its own comparison shopping service and demoting rivals; it fined Google €2.42 billion.

Why does each search engine name its crawler?

So a robots.txt file can give that crawler its own rules, such as Yeti for Naver, without affecting other engines.