A tiny AI robot is dropped into a maze. It knows nothing.
Through thousands of attempts โ hitting walls, finding dead ends,
occasionally stumbling upon the exit โ it slowly builds a mental map
and learns the perfect escape route.
""")
gr.HTML("""
๐บ๏ธ
The Maze
A grid of corridors and walls. The bot starts at
S and must reach
G.
It can only see its own position โ no map, no cheating.
๐ค
The Bot
At each step it chooses: go up, down, left, or right.
Hit a wall? Penalty. Reach the goal? Big reward!
It remembers what worked and what didn't.
๐ง
The Learning
Each attempt updates a "score table" for every
position and move. After enough tries, the bot
always picks the move with the highest score โ the optimal path.
""")
gr.HTML("""
Build a maze, pick a brain, watch it learn
The animated replay shows the final learned path after training.
""")
with gr.Row():
# โโ Controls โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
with gr.Column(scale=1, min_width=300):
gr.HTML('
Head-to-head: which brain learns fastest?
All algorithms train on the same maze with identical settings โ
the only variable is the learning strategy.
""")
gr.HTML("""
The Big Idea
The bot doesn't know anything about the maze at the start. It just knows
4 possible moves and gets a number (reward) after each step.
Negative number = bad move. Positive = good move.
The goal: find the sequence of moves that gets the most reward.
The Score Table (Q-Table)
The bot keeps a table with one row per maze cell and 4 columns (one per direction).
Each entry stores how good it thinks that move is from that cell.
At the start, everything is 0. After training, the table holds the bot's
entire learned strategy. The Q-value heatmap in the Playground shows this table visually.
Q[current_cell][move] += learning_speed ร (
reward_got + future_discount ร best_Q[next_cell] โ Q[current_cell][move]
)
Exploration vs Exploitation
Early in training, the bot tries random moves (exploration)
โ it doesn't know enough to trust its table yet. Over time, it relies more on what
it's learned (exploitation). This is controlled by epsilon (ฮต),
which starts near 1.0 (100% random) and decays toward 0 (always use best known move).
Why does reward go negative first?
Each step costs โ1 (time penalty) and hitting a wall costs โ5.
A random bot hits a lot of walls and takes forever to find the exit,
so early rewards are very negative. As it learns, fewer walls are hit and
the path shortens โ reward climbs toward 0 and eventually turns positive when
it reliably reaches the goal.
Q-Learning vs SARSA
Q-Learning always updates toward the best possible next action โ
even if it wouldn't actually take that action. SARSA updates toward
the action it will actually take. This makes SARSA more cautious near walls.
Why Monte Carlo is slow
MC waits until the episode ends before updating any scores.
On large mazes where early episodes never reach the goal,
it gets zero signal for a long time. But once it starts solving,
its estimates are very accurate.
""")
gr.HTML("""
Built with Q-Learning ยท SARSA ยท Monte Carlo ยท Gymnasium ยท Gradio
""")
if __name__ == "__main__":
demo.launch(server_name="0.0.0.0", server_port=7860, share=False, css=CSS)