Q-Learning (Reinforcement Learning)
The method by which AI does what Pavlov would have tried if he had loved tables.
Nobody shows the explorer the way. Watch it work out the route from reward and penalty alone.
The Temple of Learning
Learning Curve
Reward per episodeNo episodes yet — press start.
Agent Thoughts
What happens on the grid
This demo shows how a figure finds its way to the goal purely through reward and punishment — not from instructions. Here is what you see on the board.
- What you see
- A grid of tiles seen from above. A start flag, a diamond as the goal, plus holes as traps and walls as obstacles. The figure stands on one tile. The tiles color in like a heat map.
- What happens
- From the goal outward, the warm color spreads across the tiles: whatever lies near the diamond is marked as valuable. In each tile a small arrow points in the currently best direction. Little by little the arrows line up into one continuous path from the start flag to the goal.
- What you can do
- Start or pause the training, step through it move by move, or change the speed. The explorer tools let you show the heat map, the arrows, the Q-values and the path. Click a tile to inspect it more closely.
- What to watch for
- Nobody shows the figure the way. It learns on its own, from reward and punishment, which direction pays off. Watch how the warm color and the arrows grow into a route by themselves.
Definition
Q-Learning Definition
Analogy
Definition
Izaac Jenson enters the temple without a map or prior knowledge of its layout. Through careful exploration and learning from mistakes, he creates a "quality table" (Q-table) that captures the value of each possible action from each position. Watch as our algorithm archaeologist gradually discovers the optimal path to the Data Crystal while avoiding ancient traps.
Important Learning Parameters
- Learning Rate (α): How quickly the agent updates its knowledge. Higher values mean faster learning but less stability.
- Discount Factor (γ): How much the agent values future rewards compared to immediate ones. Values closer to 1 favor long-term planning.
- Exploration Rate (ε): The probability of trying random actions instead of the current best choice. Essential for discovering new paths.
Understanding the Visualization
The demo provides rich visual feedback for understanding the learning process:
- Q-Values: Color-coded arrows show the agent's learned preferences for each direction from each cell.
- Color Coding: Green shows positive values (good actions), red shows negative values (actions leading to traps), and blue represents the treasure.
- Agent Behavior: Observe the character's thought process in real-time as he makes decisions and updates his knowledge.
Experiment Tips
- Start with default parameters to see basic learning, then adjust one parameter at a time.
- Try high exploration rates for more diverse behavior, then lower them for more focused learning.
- Reset the environment and change parameters to see how different settings affect learning speed and quality.
- Use step-by-step mode to closely observe individual decision moments.
After watching our digital treasure hunter in action, you might be interested in what exactly happens in his "brain." Q-Learning is far more sophisticated than it appears at first glance.
The secret lies in balance. Our agent must constantly switch between two strategies: trying new things (exploration) and using proven paths (exploitation). Too much curiosity leads to chaotic behavior, too little prevents discovering better routes.
Why is Q-Learning so special?
Unlike supervised learning, where we spoon-feed the algorithm the right answers, our Q-Learning agent must figure everything out himself. He gets only minimal information:
- His current state (where am I right now?)
- Possible actions (what can I do?)
- Rewards or penalties after each action (was that good or bad?)
From these simple ingredients, the agent develops complex strategies. The fascinating part: He learns not only what is good, but also what might be good in the future.
The digital notebook: The Q-Table
Did you notice in the demo how our agent initially acts chaotically but becomes more targeted over time? This is due to his Q-table – a kind of digital notebook where he notes for each situation and action: "How good was this decision?"
When you display the Q-values in the demo, you see these numbers in real-time. Positive values mean "good idea," negative ones "better not." The more often an action in a situation leads to success, the higher its Q-value becomes.
The magic of future planning
The really clever thing about Q-Learning: It doesn't just think about the next step, but about the entire future. A path that seems laborious at the moment could lead to great later possibilities.
In the demo, you see this when the agent apparently takes "detours" – he has learned that certain positions are strategically valuable, even if they don't lead directly to the goal.
Why does this work so well?
Q-Learning combines three fundamental learning principles that you just observed in the demo:
- Trial and Error: Like a child, the agent learns through experimentation
- Reinforcement: Successful actions are preferred, bad ones avoided
- Long-term planning: Not just the next step counts, but the overall goal
The result you see above: From chaotic initial movements emerges an elegant, targeted strategy over time. And all without explicit programming of the solution path – the agent discovered it himself.
Feel free to experiment further! Change the parameters in the configuration and observe how the learning behavior changes. Each setting tells its own story about the balance between curiosity and efficiency.
1
# Q-Learning Algorithm
2
function q_learning(environment, episodes):
3
# Main function: Learns optimal policy through interaction
4
Q_table = initialize_with_zeros(states, actions)
5
α = 0.1 # Learning rate, ε = 1.0 # Exploration, γ = 0.99 # Discount
6
7
for each episode in episodes:
8
# One episode = One run from start to goal
9
state = environment.reset() # Start state
10
11
while not done:
12
# ε-greedy strategy: Explore vs. Exploit
13
if random() < ε:
14
action = choose_random_action()
15
else:
16
action = argmax(Q_table[state])
17
18
# Execute action and observe result
19
new_state, reward, done = environment.step(action)
20
# New state and reward received
21
22
# Q-value update with Bellman equation
23
current_q = Q_table[state, action]
24
max_future_q = max(Q_table[new_state])
25
target_value = reward + γ * max_future_q
26
27
# Q(s,a) ← Q(s,a) + α[r + γ·maxQ(s',a') - Q(s,a)]
28
new_q = current_q + α * (target_value - current_q)
29
Q_table[state, action] = new_q
30
31
state = new_state # Update state
32
33
# Reduce exploration rate (more exploitation)
34
ε = max(0.01, ε * 0.995) # Minimum ε = 0.01
35
36
return Q_table # Learned policy
🎯 Initialization: Blank Slate
The Q-table starts with zeros - the agent knows nothing about the environment. Each entry Q(s,a) represents the expected value of action a in state s.
Q_table = initialize_with_zeros(states, actions)
α = 0.1 # Learning rate, ε = 1.0 # Exploration, γ = 0.99 # Discount
📊 Initialize Q-Table
A table with all state-action pairs is filled with zeros. This is the agent's 'brain' where it stores its knowledge.
🔍 Exploration Phase
Initially, the agent explores the environment randomly to discover new strategies. Like a child learning through trial and error.
⚡ Exploitation Phase
With growing knowledge, the agent uses its best known strategy. The Q-values guide it to optimal decisions.
📚 Learning from Experience
Each interaction provides new information. The agent adjusts its Q-values and continuously improves its strategy.
🏆 Optimal Policy
After sufficient training, the agent has learned the optimal policy. The Q-table now contains the best actions for each state.
Check
Q-Learning Knowledge Test
What best describes the exploration-exploitation dilemma in Q-Learning?
Select one answer
1. What best describes the exploration-exploitation dilemma in Q-Learning?
2. What role does the epsilon parameter (ε) play in Q-Learning?
3. What does the Q-table store in a Q-Learning algorithm?