Mixed-Precision

TL;DR

• The Trap: Storing master weights in bfloat16 during small-learning-rate RL (lr = 5e-7) silently freezes most of the model. For a typical weight $|w| \approx 0.02$, the smallest step bf16 can resolve (half-ULP) is $6.10 \times 10^{-5}$—over $120\times$ larger than a 5e-7 Adam update. Every step rounds straight back to the original value.
• Why It Goes Unnoticed: Training does not crash or flatline. Tiny-magnitude weights ($|w| \le 10^{-4}$) and near-zero biases have much smaller ULPs and do update—causing reward and easier benchmarks (GSM8K: 35.25% → 57.01%) to climb slowly while harder reasoning (MATH500: 35.01% → 35.01%) stays frozen at the exact untrained baseline.
• The Fix: Restoring standard fp32 master weights (keeping bf16 forward/backward compute) beat 929 steps of bf16 training by Step 100, finishing +7.2 points higher on MATH500 (42.25%) and +21.3 points higher on GSM8K (78.32%), at the cost of ~7 GB/GPU extra memory and +20% step time.

The blog is co-authored with Gemini Carbon.