A duck has to find bread. Nobody tells it how. On this page the same small problem is solved three times, by three kinds of learning: evolution, a table of experience, and a neural network. All three run live in your browser. You can watch what each one learns, and you can break each one.
The three are not toys of three different fields. They are the ladder that reinforcement learning climbed, from selection to tables to the deep networks that learned to play Atari games and that walking robots are trained with today.
Evolution: learning without a learner
Thirty ducks start with random brains. Each brain is a tiny neural network of 68 numbers that turns what the duck senses into a turn and a speed. A duck that eats enough has ducklings, and each duckling gets a copy of its mother's brain with small random changes. A pike hunts them all.
No duck ever learns anything during its life. The population does.
Share of ducks with the right reflex
Every four seconds of pond time each brain gets a small test: bread on its left, then on its right, and the same with the pike. A random brain passes about a quarter of the time. Try 25× and two pikes.
Q-learning: a table of experience
Now one duck learns during its own life. It splits the world into 36 situations: where the bread is (12 directions) and how far away it is (3 rings). For each situation it keeps a score for each of five moves, from hard left to hard right. After every move it nudges that score toward what actually happened. That is Q-learning.
What the duck has learned
The whole table, drawn as a compass. The duck in the middle faces up. Each cell is a place the bread can be, near on the inside, far on the outside. The colour is the move with the best score there, strong when the duck is sure. The outlined cell is where the bread is now.
How directly it swims
100% is a straight line to the bread. The line is the average of the last 10 tries.
- Let it learn in simulation at 60× until the line flattens out.
- Turn learning off and switch to real water. There is a current, and the duck's left foot is weak.
- Turn learning back on and watch the compass change colour.
Deep Q-network: a neural network instead of a table
A table cannot cope with reeds. There are far too many situations to list. So the third duck replaces the table with a small neural network of 988 numbers. Its inputs are seven eyes that measure the distance to reeds, a nose that smells the bread, and a sense of touch. Its outputs are scores for four moves: left, straight, right and back up. It learns with the same update as the table, from a memory of its last 20,000 moves.
The smell spreads around obstacles the way it would in water, so the nose points along the shortest open path. The yellow glow on the water is that smell.
Is it getting better?
Each point is 1,000 steps. Shaded stretches had learning switched off.
What it wants to do now
The network's estimate of future reward for each move. It picks the highest. Watch Back up: it stays lowest until the duck touches a reed, then jumps.
The network right now
Senses on the left, two layers of 24 neurons, four moves on the right. Green connections excite, orange inhibit; only the stronger ones are drawn. A neuron lights up when it is active.
Three failures you can reproduce
Each lab on this page was built by watching a duck fail. These three failures are the ones that matter far beyond ducks.
Reward hacking: you get what you reward, not what you meant
The first version of Lab 3 gave the duck a nose that pointed straight at the bread and paid it for every step that brought it closer in a straight line. It learned to press itself against the reed nearest the bread and stay there, because the long way round first makes the distance worse. Nothing was wrong with the learning. The reward described a different goal than the one intended.
See it: in Lab 3, choose Bridge, switch on Straight-line nose and press Forget everything. In our test runs it found about 2 pieces of bread per 1,000 steps and hit the reeds about 40 times. With the smell that flows around obstacles it found about 5 and hit them about 3 times. The fix was to change what is rewarded, not how the duck learns.
The sim-to-real gap: the world changed, the policy did not
A policy learns the world it is trained in, including its accidents. Move it to a slightly different world and it fails in ways that look strange from outside but are perfectly consistent from inside.
See it: in Lab 2, train to over 90% in simulation, turn learning off and switch to real water. In our runs it dropped to between 20% and 35%. With learning back on it recovered to about 65%. In data work the same thing is called distribution shift.
Partial observability: you cannot learn what you cannot sense
The duck in Lab 3 only knows this one moment. It cannot tell where fleeing bread is heading, and its eyes report a pike exactly as they report a reed. More practice does not help, because the information is not in its inputs. The fix is a new sense or a memory, for example a recurrent network that carries the last few seconds forward.
See it: in Lab 3, switch on Fleeing bread or Pike after the duck has learned. The Taken counter keeps rising and the bread count stays low, however long it trains.
The three side by side
| Evolution | Q-learning | Deep Q-network | |
|---|---|---|---|
| Who learns | The population | One duck | One duck |
| Where it is stored | 68 inherited numbers per duck | A table of 36 × 5 scores | 988 network weights |
| Feedback | Survival and offspring | A reward after every move | A reward after every move |
| Learns within one life | No | Yes | Yes |
| New situations | Only those selection has met | Only those in the table | Yes, if the senses carry the information |
| Typical failure | Slow, and keeps whatever happens to survive | Sim-to-real gap | Reward hacking, no memory |
Method
- Lab 1
- Each brain has 9 inputs (direction and distance to the nearest bread and pike, a wall ahead, its own energy, a constant), 6 hidden tanh neurons and 2 outputs (turn and speed). Food gives energy, swimming costs energy, and a duck with enough energy has a duckling. Each of the 68 weights mutates with probability 0.12 by a Gaussian step of 0.3. If fewer than six ducks remain, newcomers are mutated copies of the best duck so far.
- Lab 2
- Tabular Q-learning with 36 states and 5 actions, α = 0.2, γ = 0.9. Exploration starts at 30% random moves and decays by 1.5% per try to 3%. Reward per step: −0.02, plus 0.12 per pixel closer, plus 5 for the bread. Real water adds a current of (0.5, 0.2) pixels per step, a turning bias of 0.13 radians per step and a little noise.
- Lab 3
- Double DQN with a 11–24–24–4 ReLU network, Huber loss and Adam (learning rate 0.0005), batch 32, replay memory 20,000, target network synced every 600 updates, γ = 0.95. Exploration falls from 100% to 4% over 25,000 steps. Reward per step: −0.01, plus 0.06 per pixel closer along the smell, −1 for touching a reed, +5 for the bread, −3 if the pike takes it. The smell is a shortest-path distance on a 10-pixel grid around the reeds.
Everything runs in this page in plain JavaScript, with no libraries. The test numbers quoted above come from running the same code without graphics; your runs will differ a little, since every run starts from random numbers. The Lab 3 brain is saved in your browser, so a reload does not throw away its training.
References
- Sutton, R. S. & Barto, A. G. (2018). Reinforcement Learning: An Introduction, 2nd ed. MIT Press. Free online.
- Watkins, C. J. C. H. & Dayan, P. (1992). Q-learning. Machine Learning 8, 279–292.
- Mnih, V. et al. (2015). Human-level control through deep reinforcement learning. Nature 518, 529–533.
- van Hasselt, H., Guez, A. & Silver, D. (2016). Deep reinforcement learning with double Q-learning. AAAI.
- Pollen Robotics, Microduck, a small biped trained in a physics simulator and moved to the real robot. The inspiration for this page.