Denser neq Better: Limits of On-Policy Self-Distillation for Continual Post-Training

A research paper found that on-policy self-distillation can accelerate in-domain specialization in continual post-training but has limitations in preventing forgetting and generalizing to out-of-distribution scenarios.

1sec.ai