Multi-Armed Bandit

The fundamental learning dilemma in AI: explore or exploit? — the simplest reinforcement-learning algorithm, explained at the casino

🎰 Mission: In front of you are slot machines. Each one has a hidden win probability. Find the best one with as few pulls as possible — without wasting too many coins on the bad ones.
Interactive canvas. Use arrow keys to navigate between machines, Enter or Space to pull the focused machine in manual mode. Digit keys 1 through 9 pull directly. R toggles the reveal mode.
Click a machine
Bar = estimateGlow = uncertainty● win · ○ miss

Regret curve Regret = how much reward you lost compared to the perfect choice. Flat = clever, linear = naive.

cumulative loss vs the ideal strategy
Current: 0.0 / 0 pulls
Scenarios:
👉 Press Start to run the strategy — or hit Compare all to see all four algorithms side by side.
Controls
StrategyDecides how the machine picks the next arm. "Manual" leaves the choice to you; the others explore and exploit automatically.
Total pulls0Total reward0Most played#1 (0×)% hits on best machineEnable "Reveal hidden" to see which machine is actually the best.
Run
Parameters
Share of purely random pulls. A higher ε explores more but also wastes more coins on bad machines.
0 = exploit only, 1 = full random. Classic: 0.05–0.20.
Hidden values
🔒 There are hidden values — want to see them?
Reveal hidden
Display
Bandit #1–5
Classic bandit notation: #1, #2, … #5
Advanced modes
Budget mode offCapped at 100 pulls. Then auto-pause + score card.
World stableTrue values shift slightly every 25 pulls — a strategy that has stopped exploring won’t notice.

What this view shows

Several slot machines stand side by side, each with an unknown chance of paying out. Turn by turn the method feels its way toward the best one — and you watch its choice take shape live.

What you see
A row of slot machines side by side. Each machine's window shows the latest result, with the last few pulls lined up below it as dots (● win, ○ miss). Below that a shaft: its fill is the estimated chance, and a glow shows how unsure that estimate still is.
What happens
On every turn exactly one machine lights up briefly — the one just played. Its window shows a win or a miss, the fill moves to the new estimate, and the more often it has been played, the fainter its uncertainty glows. Over time the choice lands on the same strong machine more and more often.
What you can do
Pick a strategy at the top, then hit Start or Step and watch. In manual mode you tap a machine yourself. The sliders change the speed and the number of machines, or reveal the hidden chances.
What to watch for
The tug of war between exploring and exploiting: the method has to try enough new options to find the best machine at all — but must not cling too long to a weak one.

Definition

What is a multi-armed bandit?

Analogy

You walk into a casino with five slot machines. Each pays out at a different rate — but you don't know which one is best. Do you always play the one that looks best right now? Or do you try the others, because maybe one of them pays more often?

Algorithms face this exact dilemma all the time: Which ad gets more clicks? Which treatment works better? Which recommendation gets bought? Every decision is a trade-off between exploiting (what seems to work) and exploring (what might be even better).

At first glance the casino setting feels like a gambling problem — but it's actually the simplest learning algorithm in modern AI. The Multi-Armed Bandit problem is the entry-level task of a discipline called Reinforcement Learning (RL) — the same discipline DeepMind used to train AlphaGo, and the one OpenAI used to align ChatGPT with human responses.

Machine learning has three main paradigms: Supervised learning (examples with labels — "this is a cat"), Unsupervised learning (finding patterns in unlabelled data — "these images cluster together"), and Reinforcement learning (learning from reward and punishment — "that action worked, do it again"). The Bandit here is the simplest possible RL task: no states, no consequences — just pick an action and collect a reward. That's exactly what makes it the perfect entry point.

AlphaGo & AlphaZero (game-playing agents) When AlphaGo decides which next move to explore in its game tree, it uses a variant of UCB (Upper Confidence Bound — the second strategy in this demo) at its core. The algorithm you're about to try on slot machines is the same building block that beat the world's best Go player.
ChatGPT & modern language models When an LLM is trained via RLHF (Reinforcement Learning from Human Feedback), the system has to learn which response variants humans prefer. Same question again: explore (try new phrasings) or exploit (stick with a proven phrasing). The bandit dilemma in pure form.
Netflix, Spotify, online ads Which cover art do you see? Which song gets recommended next? Which ad gets shown? Thompson Sampling (the third strategy here) is in production right now inside the recommender systems of major platforms — A/B testing is essentially a real-time bandit problem.
Robotics & autonomous driving A robot learning to grip has to decide on every attempt: "repeat the motion that worked best so far" or "try a slightly different one because it might be even better". The same question you're asking the slot machines — only with many more levers.

💡 Key insight: Whenever an AI system has to choose between options under uncertainty — and is supposed to learn which option is best — a Multi-Armed Bandit (or an extension of it) sits at the core. What you're about to see on slot machines is not casino fluff. It's a foundational building block of modern AI.

There is no single right answer. But there are three classical strategies that sit on a beautiful didactic spectrum — from "naive with dice" to "mathematically optimal".

  1. ε-greedy — the honest beginner With probability (1−ε) pull the currently best arm. With probability ε pull a random one. Simple, fast — but: ε stays constant, so even after 10,000 pulls you still waste ε percent of your coins on pure randomness.
  2. UCB1 — the self-correcting optimum Pull the arm with the highest <em>Upper Confidence Bound</em>: estimate plus uncertainty bonus. Less-tested arms get a big bonus and eventually have to be tried. Provably logarithmic regret — straight out of the textbook.
  3. Thompson Sampling — the elegant Bayes solution Keep a Beta distribution per arm (see the sparkline in the demo). Each step you draw one sample from each distribution and pull the arm with the highest sample. Probabilistic — auto-explores when estimates are uncertain, exploits when they are sharp.

The key observation: ε-greedy stays linear in the regret curve because its constant ε never stops gambling. UCB1 and Thompson flatten out logarithmically because they stop exploring as confidence grows. Hit "Compare all" — you see it in black and white after 500 pulls.

TheoryPseudocodeStep by stepFlow diagram

The dilemma: explore or exploit?

In the multi-armed bandit problem you have K actions with unknown reward distributions. At every step an algorithm picks one arm, gets a reward, and learns from it. The goal: maximise the cumulative reward over n steps — equivalently, minimise regret.

Regret(n) = n · μ* − E[Σᵢ μ_aᵢ]  — the gap between the theoretical optimum (all pulls on the best arm) and what the algorithm actually collects.

Three classical strategies

  • ε-greedy: Pull the currently best arm with probability (1−ε), a random one with ε. Simple — but linear regret because of the constant gambling.
  • UCB1: Pull the arm with the highest Upper Confidence Bound: estimate + c·√(ln t / Nᵢ). Provably O(log n) regret.
  • Thompson Sampling: Keep a Beta(α, β) distribution per arm. Draw a sample from each, play the arm with the highest sample. Bayesian and elegant.

At-a-glance comparison

Four algorithms along three axes: selection rule · regret bound · distributional assumption.

  • ε-greedy  — Selection: random with ε, else argmax μ̂ᵢ. Regret: linear O(n·ε). Assumption: none — distribution-agnostic.
  • ε-decay  — Selection: like ε-greedy, but ε(t) = max(0.02, 1/√t). Regret: sublinear O(√n). Assumption: stationary world.
  • UCB1  — Selection: argmaxᵢ (μ̂ᵢ + c·√(ln t / Nᵢ)). Regret: optimal O(log n). Assumption: bounded rewards [0,1].
  • Thompson  — Selection: argmaxᵢ Beta(αᵢ, βᵢ).sample(). Regret: optimal O(log n). Assumption: Bernoulli rewards (general: conjugate family).

Why do UCB1 and Thompson converge?

UCB1's uncertainty bonus c·√(ln t / Nᵢ) shrinks as pulls grow — it explores ever more rarely as estimates get sharper. With Thompson, the Beta distributions get narrower with every pull — samples concentrate around the true value, and the algorithm exploits automatically.

ε-greedy lacks this self-reduction. With constant ε it gambles forever at probability ε — that's the linear regret floor it can never escape. The practical fix: anneal ε over time (decaying ε-greedy).

Where do bandits show up in practice?

  • A/B testing: Adaptive tests that drop bad variants early
  • Recommender systems: Netflix, Spotify — explore new content without annoying the user
  • Online advertising: Which ad gets more clicks under a budget constraint?
  • Clinical trials: Multi-arm bandit trials assign more patients to the best treatment so far
  • News-feed ranking: Which article gets surfaced how often?

Play in the demo! Hit "Compare all" — in 500 pulls you see ε-greedy stay linear while UCB1 and Thompson flatten out logarithmically.

Check

Test your understanding

What does "regret" measure in this demo?

Select one answer

Answer Key: 1) B · 2) C · 3) B · 4) C
What if the machines changed depending on a state?
That is exactly when the bandit problem turns into a Q-learning problem: actions have consequences, the state shifts, and the agent has to learn a policy for every situation.

Plain Language

This preview is in Plain Language. The whole portal is also available in Plain Language. You can change the language at the top. Or use the button below. Our Plain Language texts are AI-assisted and editorially reviewed. They follow the core principles of easy-to-read language, but they are not certified Easy Read: a review panel of people with learning difficulties does not test them.

Preview: Demo: A slot machine with brains

What does the demo show?

This demo shows slot machines. There are 5 machines to choose. Each machine pays out differently. But: you do not know which is best. You have to figure it out. With few tries.

How to use the demo

You can play yourself. Click on a machine. You win or lose. Or you pick a strategy. The strategy plays for you. You see how fast it learns.

The big problem

Should you play the best machine? Or try a new one? This is: exploration or exploitation. Exploration means: try new things. Exploitation means: use what you know. Both matter.

What you learn

You learn: AIs often have to decide. Stay safe or try new things? This problem is everywhere. In internet recommendations. In ads. And in learning.