Title: Avoiding Collapse in Curated Synthetic Data: A Theoretical Analysis of Generative Retraining with Diverse Preferences

Abstract: The recursive retraining of generative models presents a significant representation challenge. When synthetic outputs are filtered using a static reward signal, models frequently converge on a limited range of outputs that excessively optimize for that specific objective. While previous research has indicated that such collapse is inevitable unless real data is incorporated, this study reexamines that assumption through the lens of alignment. We demonstrate that collapse can be effectively mitigated by curating data based on multiple reward functions. By formalizing the dynamics of recursive training under heterogeneous preferences, we prove that, provided certain conditions are met, the model converges to a stable distribution. This distribution distributes probability mass across various high-reward regions, thereby preserving diversity. Furthermore, the limiting distribution satisfies a weighted Nash bargaining solution, providing a rigorous formal interpretation of how values are aggregated within synthetic retraining loops.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC