ExplorableBandits

Thompson sampling, one pull at a time

Each arm is a belief about a click rate. Pull, watch the belief sharpen, and compare the regret against a fixed A/B split.

This scene needs WebGL. The regret chart, the controls and the numbers still work.

drag to orbit

Cumulative regret

Thompson sampling
UCB1
ε-greedy
A/B split

Regret after T rounds is the sum, over those rounds, of the best true rate minus the true rate of the arm that was played. It uses the true rates and never the clicks, so the lines do not jump with each reward. This is a single seeded run. An average over many seeds would be smoother, and close policies can swap places from one seed to the next.

Speed 5 / s

Beliefs shown for

Arm A, true click rate 0.30 Arm B, true click rate 0.45 Arm C, true click rate 0.60 Arm D, true click rate 0.50

Reveal true rates in the scene

Simplified: rewards are click or no click, with no context and no delay. Every policy starts from the uniform prior Beta(1, 1), and all four face the same coin flips, so in a given round an arm clicks for every policy that plays it or for none. UCB1, ε-greedy and the A/B split never use a posterior. Their ridges are the Beta posterior their own pulls would give. Ridge heights share one linear scale that shrinks once the tallest peak would leave the frame, so every ridge keeps the same area. Moving a true rate or ε replays the run to the same round on the same seed. Shift the best arm drops it 0.10 below the lowest arm (0.02 at the least) from the next round on, which breaks the fixed rates all four policies assume.

What to look for

Wide beliefs still get pulled. An arm with few pulls has a wide ridge, so now and then its draw lands to the right of the leader's and it gets another try. As the ridge narrows the draws stop reaching that far, and traffic moves to the best arm with no switch-over date.
The shape of the regret line. The A/B split pays the same price every round, so its line is straight. ε-greedy keeps paying for its random pulls. Thompson sampling and UCB1 bend toward flat as the weaker arms stop being played.
Old evidence is slow to forget. Let it run for a few hundred rounds, then shift the best arm. A ridge built from hundreds of pulls needs many more to move, and the regret lines turn upward until it does.

The article, on MediumHow Reinforcement Learning quietly runs your Recommender Systems!8 min read Next explorableThe compute-optimal ridgeScaling laws