Title: ActiveUltraFeedback: Streamlining Preference Data Generation via Active Learning

Abstract:

While Reinforcement Learning from Human Feedback (RLHF) has established itself as the prevailing method for aligning Large Language Models (LLMs), its effectiveness is frequently constrained by the substantial expense associated with gathering preference data, particularly within expert fields or low-resource settings. To mitigate this challenge, we present ACTIVEULTRAFEEDBACK, a flexible active learning framework that utilizes uncertainty metrics to selectively pinpoint the most informative responses for human annotation. This system enables the rigorous assessment of conventional response selection techniques in conjunction with two innovative approaches: DOUBLE REVERSE THOMPSON SAMPLING (DRTS) and DELTAUCB. These new methods focus on response pairs characterized by significant predicted quality disparities, building on recent evidence that such pairs offer robust signals for fine-tuning. Our experimental results indicate that ACTIVEULTRAFEEDBACK generates high-caliber datasets that drive substantial gains in downstream tasks, delivering performance that matches or exceeds static baselines while requiring only one-sixth of the annotated data. The pipeline code is accessible at https://github.com/lasgroup/ActiveUltraFeedback, and the generated preference datasets can be found at https://huggingface.co/ActiveUltraFeedback.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC