The compute-optimal ridge
Loss as a surface over model size and training tokens. Fix a compute budget and find the one split that spends it best.
This scene needs WebGL. The controls and the numbers still work.
drag to orbit
- parameters N
- tokens D
- tokens / param
- predicted loss
- vs optimum
Height is the loss above its floor E, drawn on a log scale so the valley along the curtain is visible. The tick labels on the vertical axis are the loss itself. "vs optimum" is the extra predicted loss compared with the best split of the same budget.
What to look for
- One budget, one best split. Every point on the orange curve costs the same compute. Toward one end the model is too small for its data, toward the other it sees too few tokens for its size. The curve has a single low point between them.
- The frontier is a straight line. Move the budget and the low point runs along the crease where the two halves of the surface meet. On these log axes that path is straight: parameters grow as C to the power and tokens as C to the power .
- The price of a lopsided split. Slide the bead a little and the penalty stays small. Slide it far and the penalty climbs. The readout gives the extra loss at the same compute.
- Where real models sit. Turn on the pins. They have nearly the same budget and very different splits, and the list under the switch compares each one's tokens per parameter with this fit's optimum at its budget.