How Does a Large Language Model Predict the Next Word? | How It Works

What is the short answer?

A large language model writes by repeating one small step. It reads the text so far, split into tokens, and gives every token in its vocabulary a score. A function called softmax turns those scores into probabilities.

The model then picks one token, adds it to the text and runs the step again, once for every token of the reply.

What the model has learned sits in its parameters, often billions of numbers set in training. Nothing in the loop checks a fact.

Definition: A large language model (LLM) is a neural network trained on very large amounts of text to predict the next token. At each step, its output is a probability for every token it knows.

How does text become numbers the model can use?

A tokeniser first cuts the text into tokens: whole words, word pieces, punctuation or spaces. A common word may be one token, and a rare one several. Meta says the tokeniser of its Llama 3 models, released in April 2024, has a vocabulary of 128K tokens.

Google's documentation for its Gemini models gives an approximate rule for English: a token is about 4 characters, and 100 tokens are about 60 to 80 words.

Each token then becomes an embedding, a long list of learned numbers called a vector. Tokens used in similar ways tend to end up with similar vectors, though only roughly. In an illustrative case, "cat" sits near "kitten" and far from "invoice".

In 2013, Tomas Mikolov and colleagues showed how to learn such word vectors from very large data sets and tested them on word similarity.

How does attention let the model use the words before?

In 2017, Ashish Vaswani and seven co-authors proposed the transformer, a network built on attention alone, dropping the recurrent and convolutional designs then common.

In self-attention, each token's vector is compared with those of other tokens. The comparisons become weights, and the token takes in a weighted mix of their information. A model that writes text only looks back: Hugging Face's documentation says a causal language model can attend only to tokens on its left.

Take the illustrative sentence "The glass fell off the table and it broke." At "it", attention can give "glass" a high weight, so what follows suits a broken glass.

The context window is the most tokens the model can attend to at once; text outside it is not seen. Meta, for example, trained Llama 3 on sequences of 8,192 tokens.

How does the model choose the next token?

At each step, the network gives every token in the vocabulary a raw score, a logit. Softmax raises e, about 2.718, to the power of each score and divides by the total, so the probabilities are positive and add up to 1.

A decoding rule then picks one. Greedy decoding always takes the most probable token. Sampling draws a token at random in line with the probabilities, so a token at 10% comes up about one time in ten.

Temperature, written T, divides every score before softmax:

p = exp(s ÷ T) ÷ [sum of exp(s ÷ T) over all tokens]

Here p is a token's probability and s its score. A T below 1 widens the gaps between scores, so the highest-scoring token grows likelier and the lowest-scoring one fades. Patrick von Platen's 2020 Hugging Face guide notes that as T nears zero, sampling becomes greedy choice. A T above 1 flattens the probabilities.

How is a large language model trained?

In pre-training, the model predicts the next token at every position of a huge body of text. The real next token is the answer, so no labels are needed. Meta says Llama 3 was pre-trained on over 15 trillion tokens from publicly available sources.

The error is measured by the cross-entropy loss: minus the natural logarithm of the probability given to the true token. Illustratively, a probability of 0.5 gives a loss of about 0.69, and 0.9 gives about 0.11. Training nudges the parameters to lower it.

In 2020, Jared Kaplan and colleagues reported that the loss falls as a power law as model size, data and training compute grow.

Fine-tuning then trains the model on chosen examples. For InstructGPT (March 2022), Long Ouyang and colleagues fine-tuned GPT-3 on answers written by labellers, then used their rankings of model outputs for reinforcement learning from human feedback (RLHF). In human evaluations, the 1.3 billion parameter InstructGPT was preferred to the 175 billion parameter GPT-3.

Why can a fluent answer still be wrong?

In 2025, Adam Tauman Kalai and colleagues described hallucinations as plausible yet incorrect statements that models make when they guess under uncertainty instead of admitting it. They argue that training and evaluation reward guessing over admitting doubt, as a test can reward a guess over a blank.

The model is trained to produce likely text, and a likely sentence is not always a true one. No step in next-token prediction compares a claim with a trusted record.

Knowledge also stops at a training cut-off. Meta's model card gives December 2023 for Llama 3.1's pre-training data, so the model alone knows nothing later. Text pasted into the context window can supply newer facts, but the model predicts from that text rather than checking it.

How are countries governing and building language models?

• European Union. The AI Act, Regulation (EU) 2024/1689, entered into force on 1 August 2024. Its duties for providers of general-purpose AI models apply from 2 August 2025 and include technical documentation, a copyright policy and a public summary of training content.
• China. The Cyberspace Administration of China and six other agencies issued Interim Measures for generative AI services, dated 10 July 2023 and published on 13 July 2023, in force from 15 August 2023. They cover services offered to the public in China and ask providers to take effective measures to improve training data quality and output accuracy.
• United States. The National Institute of Standards and Technology (NIST) released its voluntary AI Risk Management Framework on 26 January 2023 and a Generative AI Profile on 26 July 2024.
• United Kingdom. On 14 February 2025, the government renamed its AI Safety Institute the AI Security Institute, which evaluates risks advanced AI poses to national security and public safety.
• India. On 7 March 2024, the Cabinet approved the IndiaAI Mission, with an outlay of ₹10,371.92 crore covering, among other parts, compute of 10,000 or more GPUs and indigenous foundation models.
• France. Mistral AI, a France-based company, released Mistral 7B on 27 September 2023: an open-weight model of 7.3 billion parameters under the Apache 2.0 licence. Mistral AI says its sliding window attention handles longer text at lower cost.

What do supporters and critics argue about language models?

• Energy and water. The International Energy Agency (IEA) estimated in April 2025, as reported by Nature, that data centres used about 415 terawatt hours in 2024, about 1.5% of world electricity, and projected about 945 terawatt hours for 2030. Supporters cite its finding that AI-enabled emission cuts could offset data centres' emissions if AI is widely adopted. Critics such as Alex de Vries of VU Amsterdam warn the rise could be a serious risk to climate goals. Pengfei Li and colleagues estimated in 2023 that training GPT-3 in Microsoft's US data centres could directly evaporate 700,000 litres of fresh water.
• Copyright. A UK government report of March 2026 drew on 11,520 consultation responses. Creative industries argued a broad exception would let AI learn from their works without compensation; AI developers said exceptions would support UK AI innovation. The report says a broad exception with an opt-out is no longer the government's preferred way forward.
• Bias and harmful output. Supporters cite the InstructGPT paper's report of less toxic output after human feedback training. Critics cite Stanford's 2025 AI Index: models built to curb explicit bias still showed implicit bias, such as favouring men for leadership.
• Learning and work. Supporters cite Erik Brynjolfsson and colleagues (2023, revised 2024): an AI assistant raised customer support agents' issues resolved per hour by 15% on average. Critics cite Hamsa Bastani and colleagues (2025): among nearly 1,000 high school maths students in Turkey, those who practised with a GPT-4 tutor without guardrails scored 17% lower on exams without it than students who never had it.

How does this work with real numbers?

Both examples use illustrative numbers; only the words-per-token rule comes from a source.

How does temperature change the choice of the next word?

Suppose the text so far is "The cat sat on the", with only three possible tokens and illustrative scores: "mat" 2.0, "sofa" 1.0 and "moon" 0.1.

At T = 1, exp(2.0) ≈ 7.39, exp(1.0) ≈ 2.72 and exp(0.1) ≈ 1.11. The total is about 11.21, so "mat" gets 7.39 ÷ 11.21 ≈ 0.66.

Table: Illustrative probabilities at two temperatures

At T = 0.5, the scores double to 4.0, 2.0 and 0.2, which sharpens the choice towards "mat". Greedy choice always picks "mat". Sampling picks "moon" about one time in ten at T = 1, but one in fifty at T = 0.5.

How much English fits in a context window?

Take an illustrative window of 8,000 tokens and Google's 60 to 80 English words per 100 tokens. The window holds roughly 8,000 × 0.6 = 4,800 to 8,000 × 0.8 = 6,400 words.

The prompt and the reply share the window, since each new token joins the text. A pasted 1,000-word essay takes about 1,000 ÷ 0.8 = 1,250 to 1,000 ÷ 0.6 ≈ 1,670 tokens, leaving roughly 6,330 to 6,750 for the answer. All figures are approximate.

Where can the student go deeper in World 101?

This page follows a language model one token at a time. The World 101 subject Generative AI & Large Language Models takes the wider view: what these systems can make, how people use them and the questions they raise for knowledge, work, schools and education.

Three related subjects go further. The Math Behind Machine Learning covers the probability, vectors and step-by-step optimisation behind softmax, embeddings and a falling loss. Artificial Intelligence & the Mind asks whether predicting text well is the same as understanding it. AI Governance looks at how rules such as the EU AI Act and China's interim measures set duties for the companies that make and offer these systems.

Sources

• T. Mikolov and others, word vectors, arXiv:1301.3781, January 2013.
• A. Vaswani and others, "Attention Is All You Need", arXiv:1706.03762, June 2017.
• J. Kaplan and others, scaling laws, arXiv:2001.08361, January 2020.
• P. von Platen, Hugging Face blog, March 2020; Hugging Face documentation, read 9 November 2026.
• L. Ouyang and others (InstructGPT), arXiv:2203.02155, March 2022.
• Google AI Studio documentation on tokens, read 9 November 2026.
• Meta, Llama 3 announcement, 18 April 2024; Llama 3.1 model card, 23 July 2024.
• A. T. Kalai and others, arXiv:2509.04664, September 2025.
• European Commission, AI Act pages, 1 August 2025 and 19 October 2026; Regulation (EU) 2024/1689.
• Cyberspace Administration of China, order No. 15, 13 July 2023 (Chinese text); Future of Privacy Forum, 22 April 2024; China Daily, 14 July 2023.
• NIST, AI Risk Management Framework page (2023; profile, 2024), read 9 November 2026.
• UK government, 14 February 2025 (AI Security Institute) and 18 March 2026 (copyright report).
• Press Information Bureau, IndiaAI Mission, 7 March 2024.
• Mistral AI, Mistral 7B, 27 September 2023; SiliconANGLE (Paris-based Mistral AI), 27 September 2023.
• Nature via Scientific American, 10 April 2025; PV Tech, 11 April 2025 (IEA, Energy and AI, projection to 2030).
• P. Li and others, arXiv:2304.03271, April 2023, revised March 2025.
• Stanford HAI, AI Index Report 2025.
• E. Brynjolfsson, D. Li and L. Raymond, arXiv:2304.11771, April 2023, revised November 2024; H. Bastani and others, PNAS, 2025.

Common errors and misconceptions

• Misconception: The model looks answers up in a store of saved text. Correct: It computes probabilities for the next token from its learned parameters; no step checks a fact.
• Misconception: A token is the same as a word. Correct: Tokens are often pieces of words; Google's rule of thumb is about 60 to 80 English words per 100 tokens.
• Misconception: The model always picks the most likely word. Correct: That is greedy decoding; systems can also sample, with temperature setting how spread out choices are.
• Misconception: A trained model knows about recent events. Correct: Its training data stops at a cut-off; Meta gives December 2023 for Llama 3.1's pre-training data.
• Misconception: A fluent, confident answer is probably true. Correct: Fluency comes from predicting likely text; hallucinations are plausible but incorrect.

Key takeaways

• An LLM writes one token at a time: it scores every token in its vocabulary, turns the scores into probabilities with softmax and picks one.
• Each token becomes an embedding, a vector of learned numbers, and tokens used in similar ways tend to get similar vectors.
• Self-attention, the core of the 2017 transformer, lets each token in a text-writing model weigh earlier tokens, but only those inside the context window.
• Temperature divides the scores before softmax: a lower temperature sharpens the choice, and a higher one spreads it out.
• Pre-training lowers the loss on next-token prediction; fine-tuning and reinforcement learning from human feedback then shape the model to follow instructions.
• The European Union, China, the United States, the United Kingdom and India have each set out laws, frameworks, institutes or programmes for AI.

Test yourself

What does the softmax function do?

It turns the raw scores for every token into probabilities that are positive and add up to 1.

In the worked example, what happens to "mat" when T falls from 1 to 0.5?

Its probability rises from about 0.66 to about 0.86, as dividing by 0.5 doubles the gaps between scores.

Why can a model not use text outside its context window?

Self-attention only reaches tokens inside the window, so anything beyond it is never seen during prediction.

What did the InstructGPT paper of 2022 report?

In human evaluations, a 1.3 billion parameter model trained with human feedback was preferred to the 175 billion parameter GPT-3.

Why do language models hallucinate, according to Kalai and colleagues?

Training and evaluation reward guessing over admitting uncertainty, so models produce plausible yet incorrect statements.

What must general-purpose AI model providers do under the EU AI Act?

Duties applying from 2 August 2025 include drawing up technical documentation, having a copyright policy and publishing a summary of training content.