Back to Learning Area

Sampling & Temperature

The mini-roulette after every token — and the dials you can turn.

  • Architectures
  • 10 min
  • Intermediate

Plain Language

This preview is in Plain Language. The whole portal is also available in Plain Language. You can change the language at the top. Or use the button below. Our Plain Language texts are AI-assisted and editorially reviewed. They follow the core principles of easy-to-read language, but they are not certified Easy Read: a review panel of people with learning difficulties does not test them.

Preview: How AI chooses the next word

Logits and softmax

A language model calculates a number for every word. These numbers are called logits. Logits show: which word is more likely? Softmax converts these logits into probabilities.

The temperature

The temperature is a dial for language models. A high temperature makes text more creative. A low temperature makes text more predictable. At temperature 0, the model always chooses the most likely word.

Sampling

Sampling means: the model chooses the next word randomly. While doing this, the model pays attention to probabilities. More likely words appear more often. This is how naturally sounding texts emerge.

You know that a language model predicts the next token. But between the model's internal computation and the word that appears on your screen lies a critical decision layer that most users never see. Here, roughly 100,000 candidates are scored, ranked, and one is selected — and the rules governing this selection determine whether the output is safe, creative, repetitive, or nonsensical. This article opens the black box of LLM settings: Softmax, Sampling, and Temperature. No advanced math required — just the willingness to understand why that "Temperature" slider exists and what it actually controls.

Softmax — From Raw Scores to Probabilities

After the forward pass, a Transformer outputs a raw score for every token in its vocabulary — a so-called logit. With a vocabulary of 100,000 tokens (depending on the model, it's roughly 32,000 to 200,000), that means 100,000 numbers. These logits are abstract values with no direct interpretation: a logit of 4.2 for "food" and -3.1 for "algebra" tells you that "food" is preferred, but not how likely it ends up being — and the numbers don't form a usable probability distribution. This is exactly where Softmax comes in.

Definition

Softmax

Analogy

Imagine a cooking competition with four judges who award points on an open scale, including negative values and decimals. The raw scores say little on their own. To pick a winner, a coordinator converts all ratings into percentages: Dish A gets 45%, Dish B 30%, Dish C 20%, Dish D 5% — totaling exactly 100%. Softmax is that coordinator: it takes arbitrarily scaled numbers and produces a fair, normalized ranking.

Example

The analogy has limits: jury ratings have humanly interpretable meaning — a "7 out of 10" conveys something. Logits have no intrinsic meaning. Also, a cooking competition might evaluate 10 dishes; an LLM evaluates 100,000 tokens simultaneously.
  1. 1. LogitsRaw scores: Food = 4.2 | Toy = 2.8 | Bed = 0.5 | Algebra = -3.1
  2. 2. Exponentiatee to the power of logit: 66.7 | 16.4 | 1.6 | 0.05
  3. 3. NormalizeDivide by sum (84.8)
  4. 4. ProbabilitiesFood = 78.6% | Toy = 19.4% | Bed = 1.9% | Algebra = 0.1%

Concrete walkthrough: Prompt: "The dog eats its..."

Logits: "Food" = 4.2, "Toy" = 2.8, "Bed" = 0.5, "Algebra" = -3.1. After Softmax: "Food" = 78.6%, "Toy" = 19.4%, "Bed" = 1.9%, "Algebra" = 0.1%. The exponential function amplifies the gap — the difference of 1.4 logit points between "Food" and "Toy" becomes a ratio of roughly 4:1. "Algebra" with its negative logit is virtually eliminated.

Note

The principle: Softmax turns winners into bigger winners and losers into bigger losers.

Interactive: The Softmax Formula Step by Step

The formula behind the four steps above, already including the temperature T from the section further down. The example uses three logits, 2, 1 and 0.5, at T = 1: dividing by T then changes nothing, and a gap of one logit point between the first two becomes 63% to 23%, roughly 2.7 to 1. That is how Softmax amplifies differences.

Sampling — Rolling Dice by Rules

Softmax has produced a probability distribution. Now the system must select a single token. There are two fundamental strategies for this.

Greedy Decoding

Always picks the token with the highest probability. Deterministic in principle — same input, same output; cloud APIs, however, don't guarantee identical answers even at T=0. Problem: repetitive, generic text, infinite loops ("The dog is a dog is a dog...").

Stochastic Sampling

Draws randomly from the distribution, weighted by probability. "Food" wins most of the time, but "Toy" wins sometimes too — just like in natural human language, where we don't mechanically choose the most predictable word.

Imagine a wheel of fortune divided into differently sized segments: "Food" covers 78.6%, "Toy" 19.4%, "Bed" 1.9%. Greedy Decoding means always pointing at the largest segment without spinning the wheel. Stochastic Sampling means spinning the wheel and accepting where it lands. The largest segment wins most of the time — but not always. And that "not always" is what makes text feel human.

Note

Test the prompt "The dog eats its..." five times:

Greedy (T=0): "food", "food", "food", "food", "food" — identical every time.

Sampling (T=1): "food", "toy", "food", "bed", "food" — "food" dominates, but variety emerges naturally.

The greedy output is technically "correct" (food is the most probable), but linguistically dead. The sampled output reflects the actual distribution and sounds more natural.

Temperature & Top-P — The Creativity Controls

Temperature (T) modifies the Softmax function by dividing all logits by T before exponentiation. This doesn't change what the model "thinks" — the logits remain identical. It only changes how the final selection is weighted.

T = 0

Deterministic

Always the most probable token. Safe but repetitive and monotonous.

"...promising. Artificial intelligence will continue to..."
T = 0.7

Balanced

Moderate variation. Natural-sounding, good balance between safety and creativity.

"...a balancing act between innovation and responsibility. While..."
T = 1.5

Creative

Higher variation. Unusual phrasing, unstable in longer texts.

"...a kaleidoscopic dance between silicon dreams and the fragile melody of human intuition..."
T = 2.5

Chaotic

Distribution strongly flattened. Structure breaks down, output becomes unusable.

"...banana quantum physics umbrella strategically dinosaur..."

Temperature is like the sobriety level at a dinner party. At T=0 (stone-cold sober), every guest only says the safest and most obvious thing — predictable and boring. At T=0.7 (one glass of wine), people get wittier and more creative. At T=2.0 (heavily drunk), they ramble incoherently. Important: alcohol actually impairs thinking. Temperature does NOT change the model's computation — the logits stay identical at every Temperature. Only the selection weighting changes.

Top-P (also called Nucleus Sampling) is a complementary filter: instead of considering all 100,000 tokens, it sums probabilities from the top and discards everything that follows once a threshold is reached (e.g., P=0.9 means 90% cumulative probability). This dynamically adjusts how many tokens qualify as candidates. A similar filter is Top-K, which simply passes only the top K candidates — no matter how much probability falls on them. Top-P is the bouncer at the party: no matter how drunk the crowd gets, the bouncer blocks the worst contributions.

Not every model still lets you turn the dial: on many of the newest models from Anthropic (from Claude Opus 4.7 on), OpenAI (whenever the model uses reasoning) and Google (from Gemini 3.6 on), you can no longer change Temperature through the API (as of November 2026); there, you steer style and creativity through the prompt. You can still turn the dial freely mainly on open and older models. The scale isn't the same everywhere either: OpenAI allows values from 0 to 2, older Claude models only 0 to 1. In practice, the following Temperature guidelines have become established for different tasks:

Code Generation T = 0 – 0.3
Customer Support T = 0.5 – 0.7
Creative Writing T = 0.8 – 1.2
Brainstorming T = 1.0 – 1.5

Caution

Temperature Does NOT Prevent Hallucinations

Hallucinations arise because the model produces statistically plausible continuations, even where it does not reliably know the facts. At T=0, the model picks its most confident answer — but confidence and correctness are not the same thing. A model can output a factually wrong statement with 99% probability. Temperature controls randomness, not accuracy. To reduce hallucinations, you need Retrieval-Augmented Generation (RAG), grounding, or fine-tuning — not a Temperature slider.

Interactive: Try Token Sampling

Now it is your turn: set the temperature and roll for tokens. At low temperature, the most probable token wins almost every time. At high temperature, the results spread more evenly. Click multiple times and observe how the actual distribution approaches the theoretical one.

An LLM has generated the beginning "Das Wetter heute ___" ("The weather today ___") and computes probabilities for the next word. The most natural continuation is "ist" ("is") — but the temperature determines whether the model always picks the safe choice or dares more unusual continuations.

Controls
Standard (T≈1.0): The original logit probabilities are used. Balance between precision and variety.
Samples0Most frequent–Expectedist · 73 %

Probability Distribution (at T=1.0)

ist
73.3%
war
13.4%
wird
8.1%
soll
2.7%
bleibt
1.5%
kann
1.0%

Results (0 Samples)

No samples yet — click "Sample token"

Start the Experiment

Click "Sample token" to see how the LLM samples at the current temperature. Observe how the distribution of results approaches the theoretical probability with more samples.

The Softmax formula is: P(Token_i) = e^(Logit_i) / Sum(e^(Logit_j)). The exponential function e^x has two crucial properties: it makes all values positive (necessary for probabilities) and it amplifies differences non-linearly. With our example logits (4.2 and 2.8): e^4.2 = 66.7 and e^2.8 = 16.4. The logit difference of 1.4 points becomes a ratio of 4:1. For Temperature, the formula extends to: P(Token_i) = e^(Logit_i/T) / Sum(e^(Logit_j/T)). At T<1, the gaps between logits are amplified before exponentiation (sharper distribution). At T>1, they are compressed (flatter distribution).

Greedy Decoding can trap the model in infinite loops: when a repetitive pattern starts ("The dog is..."), the most probable next token is exactly the one that continues the loop. Repetition Penalty counteracts this by artificially downweighting the logits of previously used tokens. In the common variant from the CTRL paper (2019), used for example by Hugging Face Transformers, 1.0 means no penalty (OpenAI instead uses frequency_penalty and presence_penalty, with 0 as the neutral value); a typical value is 1.1 to 1.3 — enough to reduce repetitions without degrading text quality. Values that are too high (>1.5) can force the model to use awkward synonyms.

Note

Takeaways

  • Softmax is the translator between the model's internal world (abstract logit values) and the human-readable world (probabilities that sum to 100%) — without it, no token selection is possible.
  • Greedy Decoding (T=0) is safe but dead — it always picks the most probable token and can trap the model in repetitive loops. Stochastic Sampling lets lower-ranked candidates win too and restores the natural unpredictability of human language.
  • Temperature reshapes the probability curve, not the model's knowledge — the model "thinks" the same thing at every Temperature. Low T makes it conservative, high T makes it experimental. But T=0 doesn't prevent hallucinations, because a confidently wrong answer is still the top token.

The sampling concepts from this article apply beyond text generation. In Diffusion Models, analogous stochastic processes control image generation — a topic covered in the next article in this series.

Check

Checkpoint: Sampling & Temperature

Can you explain this in your own words? Tick what you've got.

0 of 3 done

Check

Quiz: Sampling & Temperature

Not completed

A developer receives the raw values [5.0, 3.0, 1.0, -2.0] from a model and needs to select a token. Why can't they use these numbers directly, and what transformation is needed?

Select one answer

Answer Key: 1) B · 2) B · 3) A · 4) B · 5) B · 6) C