The mini-roulette after every token — and the dials you can turn.
Architectures 10 min Intermediate June 15, 2026
You know that a language model predicts the next token. But between the model's internal computation and the word that appears on your screen lies a critical decision layer that most users never see. Here, 100,000 candidates are scored, ranked, and one is selected — and the rules governing this selection determine whether the output is safe, creative, repetitive, or nonsensical. This article opens the black box of LLM settings: Softmax, Sampling, and Temperature. No advanced math required — just the willingness to understand why that "Temperature" slider exists and what it actually controls.
Softmax — From Raw Scores to Probabilities
After the forward pass, a Transformer outputs a raw score for every token in its vocabulary — a so-called logit. With roughly 100,000 tokens in the vocabulary, that means 100,000 numbers. These logits are abstract values with no direct interpretation: a logit of 4.2 for "food" and -3.1 for "algebra" tells you that "food" is preferred, but not by how much — and the numbers don't form a usable probability distribution. This is exactly where Softmax comes in.
Softmax
AnalogyDefinition
Imagine a cooking competition with four judges who use completely different scales. One judge gives 0 to 10, another -50 to 200, a third uses decimals. The raw scores aren't comparable. To pick a winner, a coordinator converts all ratings into percentages: Dish A gets 45%, Dish B 30%, Dish C 20%, Dish D 5% — totaling exactly 100%. Softmax is that coordinator: it takes arbitrarily scaled numbers and produces a fair, normalized ranking.
Example
The analogy has limits: jury ratings have humanly interpretable meaning — a "7 out of 10" conveys something. Logits have no intrinsic meaning. Also, a cooking competition might evaluate 10 dishes; an LLM evaluates 100,000 tokens simultaneously.
Analogy:
Imagine a cooking competition with four judges who use completely different scales. One judge gives 0 to 10, another -50 to 200, a third uses decimals. The raw scores aren't comparable. To pick a winner, a coordinator converts all ratings into percentages: Dish A gets 45%, Dish B 30%, Dish C 20%, Dish D 5% — totaling exactly 100%. Softmax is that coordinator: it takes arbitrarily scaled numbers and produces a fair, normalized ranking.
Example
The analogy has limits: jury ratings have humanly interpretable meaning — a "7 out of 10" conveys something. Logits have no intrinsic meaning. Also, a cooking competition might evaluate 10 dishes; an LLM evaluates 100,000 tokens simultaneously.
Definition:
A mathematical function that converts arbitrarily scaled logits (raw values from the neural network) into a probability distribution that sums to 1.0. Two steps: first, each logit is exponentiated (all values become positive, differences are amplified), then each result is divided by the sum of all exponentiated values (normalization to 100%).
1
Logits Raw scores: Food = 4.2 | Toy = 2.8 | Bed = 0.5 | Algebra = -3.1
2
Exponentiate e to the power of logit: 66.7 | 16.4 | 1.6 | 0.04
3
Normalize Divide by sum (84.8)
4
Probabilities Food = 78.6% | Toy = 19.4% | Bed = 1.9% | Algebra = 0.1%
Concrete walkthrough: Prompt: "The dog eats its..."
Logits: "Food" = 4.2, "Toy" = 2.8, "Bed" = 0.5, "Algebra" = -3.1. After Softmax: "Food" = 78.6%, "Toy" = 19.4%, "Bed" = 1.9%, "Algebra" = 0.1%. The exponential function amplifies the gap — the difference of 1.4 logit points between "Food" and "Toy" becomes a ratio of roughly 4:1. "Algebra" with its negative logit is virtually eliminated.
The principle: Softmax turns winners into bigger winners and losers into bigger losers.
Interactive: Why Softmax Amplifies Differences
Softmax computes e to the power of each logit. Move the slider and observe how the exponential function dramatically dominates all other functions as values grow. This exact effect is why even small logit differences turn into large probability differences.
120
f(x) = 11
f(x) = x100
f(x) = x²10.000
f(x) = 2ˣ1.073.741.824
Moderate Input
At n=100, the difference becomes visible: O(n²) requires 10.000 operations, while O(n) needs only 100. O(log n) needs just 6.6 — that's 15x less than O(n).
Ratio to O(n)
Complexity
Operations
Factor vs. O(n)
f(x) = 1
1
100x faster
f(x) = x
100
1x (Reference)
f(x) = x²
10.000
100x slower
f(x) = 2ˣ
1.073.741.824
10737418x slower
Sampling — Rolling Dice by Rules
Softmax has produced a probability distribution. Now the system must select a single token. There are two fundamental strategies for this.
Greedy Decoding
Always picks the token with the highest probability. Deterministic — same input, same output. Problem: repetitive, generic text, infinite loops ("The dog is a dog is a dog...").
Stochastic Sampling
Draws randomly from the distribution, weighted by probability. "Food" wins most of the time, but "Toy" wins sometimes too — just like in natural human language, where we don't mechanically choose the most predictable word.
Imagine a wheel of fortune divided into differently sized segments: "Food" covers 78.6%, "Toy" 19.4%, "Bed" 1.9%. Greedy Decoding means always pointing at the largest segment without spinning the wheel. Stochastic Sampling means spinning the wheel and accepting where it lands. The largest segment wins most of the time — but not always. And that "not always" is what makes text feel human.
The greedy output is technically "correct" (food is the most probable), but linguistically dead. The sampled output reflects the actual distribution and sounds more natural.
2022 Products
ChatGPT Marks a Turning Point in AI Adoption
The moment AI became accessible to everyone and a new era began. On November 30, 2022, OpenAI released ChatGPT as a free research preview — without major marketing, with few expectations. What followed exceeded all forecasts: after 5 days ChatGPT reached one million users, after two months 100 million — at the time the fastest user growth any consumer application had ever achieved (surpassed in July 2023 by Meta's Threads). Built on GPT-3.5, ChatGPT gave a broad audience direct access to a powerful AI for the first time, without any technical barriers. Kevin Roose of the New York Times called it the 'best AI chatbot ever released to the public.' ChatGPT democratized artificial intelligence, turning a research field into an everyday tool. This release marked the beginning of the current generative AI wave.
Temperature & Top-P — The Creativity Controls
Temperature (T) modifies the Softmax function by dividing all logits by T before exponentiation. This doesn't change what the model "thinks" — the logits remain identical. It only changes how the final selection is weighted.
T = 0
Deterministic
Always the most probable token. Safe but repetitive and monotonous.
"...promising. Artificial intelligence will continue to..."
T = 0.7
Balanced
Moderate variation. Natural-sounding, good balance between safety and creativity.
"...a balancing act between innovation and responsibility. While..."
T = 1.5
Creative
Higher variation. Unusual phrasing, unstable in longer texts.
"...a kaleidoscopic dance between silicon dreams and the fragile melody of human intuition..."
T = 2.5
Chaotic
Distribution nearly uniform. Structure breaks down, output becomes unusable.
Temperature is like the sobriety level at a dinner party. At T=0 (stone-cold sober), every guest only says the safest and most obvious thing — predictable and boring. At T=0.7 (one glass of wine), people get wittier and more creative. At T=2.0 (heavily drunk), they ramble incoherently. Important: alcohol actually impairs thinking. Temperature does NOT change the model's computation — the logits stay identical at every Temperature. Only the selection weighting changes.
Top-P (also called Nucleus Sampling) is a complementary filter: instead of considering all 100,000 tokens, it sums probabilities from the top and discards everything below a threshold (e.g., P=0.9 means 90% cumulative probability). This dynamically adjusts how many tokens qualify as candidates. A similar filter is Top-K, which simply passes only the top K candidates — regardless of their probabilities. Top-P is the bouncer at the party: no matter how drunk the crowd gets, the bouncer blocks the worst contributions.
In practice, the following Temperature guidelines have become established for different tasks:
Code Generation T = 0 – 0.3
Customer Support T = 0.5 – 0.7
Creative Writing T = 0.8 – 1.2
Brainstorming T = 1.0 – 1.5
Temperature Does NOT Prevent Hallucinations
Hallucinations arise because the model's training data creates statistical patterns around incorrect facts. At T=0, the model picks its most confident answer — but confidence and correctness are independent of each other. A model can output a factually wrong statement with 99% probability. Temperature controls randomness, not accuracy. To reduce hallucinations, you need Retrieval-Augmented Generation (RAG), grounding, or fine-tuning — not a Temperature slider.
Interactive: Try Token Sampling
Now it is your turn: set the temperature and roll for tokens. At low temperature, the most probable token wins almost every time. At high temperature, the results spread more evenly. Click multiple times and observe how the actual distribution approaches the theoretical one.
An LLM has generated the beginning "Das Wetter heute ___" ("The weather today ___") and computes probabilities for the next word. The most natural continuation is "ist" ("is") — but the temperature determines whether the model always picks the safe choice or dares more unusual continuations.
0.1 (focused)2.0 (creative)
Standard (T≈1.0): The original logit probabilities are used. Balance between precision and variety.
Probability Distribution (at T=1.0)
ist
73.3%
war
13.4%
wird
8.1%
soll
2.7%
bleibt
1.5%
kann
1.0%
Results (0 Samples)
No samples yet — click "Sample token"
Start the Experiment
Click "Sample token" to see how the LLM samples at the current temperature. Observe how the distribution of results approaches the theoretical probability with more samples.
The Math Behind Softmax
The Softmax formula is: P(Token_i) = e^(Logit_i) / Sum(e^(Logit_j)). The exponential function e^x has two crucial properties: it makes all values positive (necessary for probabilities) and it amplifies differences non-linearly. With our example logits (4.2 and 2.8): e^4.2 = 66.7 and e^2.8 = 16.4. The logit difference of 1.4 points becomes a ratio of 4:1. For Temperature, the formula extends to: P(Token_i) = e^(Logit_i/T) / Sum(e^(Logit_j/T)). At T<1, the gaps between logits are amplified before exponentiation (sharper distribution). At T>1, they are compressed (flatter distribution).
Repetition Penalty — Why Greedy Creates Loops
Greedy Decoding can trap the model in infinite loops: when a repetitive pattern starts ("The dog is..."), the most probable next token is exactly the one that continues the loop. Repetition Penalty counteracts this by artificially downweighting the logits of previously used tokens. A typical value is 1.1 to 1.3 — enough to reduce repetitions without degrading text quality. Values that are too high (>1.5) can force the model to use awkward synonyms.
Takeaways
Softmax is the translator between the model's internal world (abstract logit values) and the human-readable world (probabilities that sum to 100%) — without it, no token selection is possible.
Greedy Decoding (T=0) is safe but dead — it always picks the most probable token and traps the model in repetitive loops. Stochastic Sampling lets lower-ranked candidates win too and restores the natural unpredictability of human language.
Temperature reshapes the probability curve, not the model's knowledge — the model "thinks" the same thing at every Temperature. Low T makes it conservative, high T makes it experimental. But T=0 doesn't prevent hallucinations, because a confidently wrong answer is still the top token.
The sampling concepts from this article apply beyond text generation. In Diffusion Models, analogous stochastic processes control image generation — a topic covered in the next article in this series.
Checkpoint: Sampling & Temperature
Why can a language model not use its raw values (logits) directly as probabilities — and what does Softmax do with them?
Using the wheel of fortune analogy, explain the difference between Greedy Decoding and stochastic sampling.
Why does setting Temperature to 0 (Greedy Decoding) not prevent hallucinations?
Quiz: Sampling & Temperature
Question 1 / 6
Not completed
A developer receives the raw values [5.0, 3.0, 1.0, -2.0] from a model and needs to select a token. Why can't they use these numbers directly, and what transformation is needed?
1. A developer receives the raw values [5.0, 3.0, 1.0, -2.0] from a model and needs to select a token. Why can't they use these numbers directly, and what transformation is needed?
☐ A) The numbers are too large — they need to be divided to a scale of 0 to 1.
☐ B) The raw values (logits) don't form a probability distribution — they don't sum to 1.0 and have no direct meaning as probabilities. Softmax converts them into a normalized distribution.
☐ C) The negative values need to be removed first, then the remaining numbers are usable.
☐ D) The numbers can be interpreted directly as probabilities — no transformation needed.
2. Why does Greedy Decoding (Temperature=0) frequently produce repetitive text?
☐ A) The model doesn't have enough vocabulary.
☐ B) It always picks the most probable token, which reinforces repetitive patterns — once a loop starts, the most probable next token is exactly the one that continues the loop.
☐ C) The training data was repetitive.
☐ D) The model repeats itself intentionally for emphasis.
3. A developer is building a customer support chatbot. At Temperature=0, the bot sounds robotic. At Temperature=2.0, the responses become incomprehensible. What setting would you recommend?
☐ A) Temperature=0.5 to 0.7 with Top-P=0.9 — moderate variation with a safety filter against extreme tokens.
☐ B) Keep Temperature=0 — users will get used to it.
☐ C) Temperature=3.0 with Top-K=5 — maximum creativity.
☐ D) Remove Temperature entirely and only use Top-K=1.
4. A medical startup sets Temperature=0 for its AI diagnostic assistant to guarantee factual correctness. The system still outputs incorrect medical information with high confidence. Analyze why this approach fails.
☐ A) Temperature=0 guarantees accuracy — the errors must come from a bug in the code.
☐ B) Temperature controls randomness, not accuracy. The model can have maximum confidence in incorrect facts from its training data. RAG with verified medical databases or domain-specific fine-tuning would address the actual problem.
☐ C) Temperature should be set to exactly 0.5 for medical applications.
☐ D) The model needs more parameters to be accurate.
5. You're developing a creative writing assistant. At which Temperature setting and why do you get the best results?
☐ A) T=0 — creativity requires precision and reproducibility.
☐ B) T=0.8 to 1.2 — enough variation for unusual phrasing, but still enough coherence for connected text.
☐ C) T=3.0 — the higher the Temperature, the more creative the output.
☐ D) T=0.1 with Top-K=100 — low Temperature plus many candidates yields controlled creativity.
6. A model outputs the distribution Food=78.6%, Toy=19.4%, Bed=1.9%, Algebra=0.1% at T=1. How does this distribution change when you lower the Temperature to 0.5?
☐ A) All probabilities are halved.
☐ B) The distribution becomes flatter — Algebra gets significantly more probability.
☐ C) The distribution becomes sharper — Food dominates even more, Algebra drops even further. The top token gains disproportionately.
☐ D) The distribution stays the same, only the selection speed changes.
Answer Key: 1) B · 2) B · 3) A · 4) B · 5) B · 6) C