Paper
When more self-play makes an agent worse.
Empirical instability in MuZero-style training.
When More Self-Play Makes an Agent Worse: Empirical Instability in MuZero-Style Training
Clark Whitehead · October 2026
In a MuZero-style agent trained in a custom combat environment, head-to-head performance commonly peaks after roughly 24,000–32,000 self-play games and then declines, even while training loss keeps improving. Lower learning rates, wider models and longer training do not consistently prevent it; continuing a strong checkpoint against a pool of historical opponents is a preliminary stabilization result.
Working manuscript — preliminary, not peer reviewed. It does not establish a causal mechanism.