Title: Escaping the Mode Lottery: Multi-Response Training Improves Language Model Generalization

Abstract:

Current practices in language model fine-tuning typically associate each prompt with only one response, despite the fact that numerous prompts can legitimately generate multiple completions. This approach effectively transforms a multi-modal conditional distribution into a single-sample perspective, a phenomenon we term the "mode lottery." In this scenario, training focuses on a limited subset of plausible modes while neglecting others. To address this, we investigate Multi-Response Training (MRT), a method that preserves several responses for each prompt, and provide a theoretical framework explaining its benefits and optimal conditions for use.

Our primary finding is that prompts and responses serve as distinct statistical resources. Increasing the number of prompts helps reduce uncertainty regarding the input distribution, whereas adding more responses decreases uncertainty about the conditional output distribution. This distinction creates a variance-budget trade-off that predicts when retaining multiple responses is advantageous. The analysis reveals diminishing returns when uncertainty at the prompt level becomes dominant and clarifies why large, redundant corpora can implicitly mimic multi-response effects.

We also examine response selection strategies, establishing that Random-K-of-N serves as the unbiased baseline for distributional fine-tuning. In contrast, relying solely on reward-based selection can lead to mode collapse, whereas a submodular quality-diversity objective offers a more efficient alternative with strong theoretical backing. Controlled simulations confirm these predictions regarding variance and selection effects, highlighting a critical failure mode: reward-only selection can generate gradients that are misaligned with the true objective.

Testing across both structured and real-world datasets, including a novel multi-prompt, multi-response benchmark, MRT consistently enhances distributional generalization. The most significant improvements occur in environments characterized by high response diversity and low prompt redundancy. Ultimately, MRT redefines response multiplicity not merely as a heuristic, but as a data-allocation problem with clear statistical justification: when responses are inexpensive and diverse, retaining multiple options is a theoretically sound decision.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC