Title: SenseJudge: A Human-Centric, Preference-Driven Judgment Framework

Abstract:

The utilization of Large Language Models (LLMs) as evaluators for assessing model responses across diverse contexts is gaining widespread acceptance. Nevertheless, current evaluation methodologies frequently depend on static, trained judges anchored in fixed preference datasets. This rigidity often fails to account for the spectrum of user preferences and hinders adaptability to the dynamic nature of real-world human-AI interactions. To overcome these challenges, we introduce SenseJudge, a flexible judgment framework tailored by human preferences, accompanied by SenseBench—a comprehensive and demanding benchmark for instruction-following derived from authentic, multi-turn conversational data. We validated our automatic judgment framework and benchmark through two primary applications: employing LLMs as personalized evaluators and conducting model rankings. Our extensive experimental results indicate that SenseJudge outperforms existing judgment methods and models in the context of personalized LLM judging. Furthermore, it produces model rankings that closely mirror human judgment. Through additional analyses of position bias and consistency, as well as ablation studies, we confirmed the robustness of the SenseJudge framework.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC