Title: Casual as an Anchor: Resolving Supervision Misalignment in Formality Transfer Dataset

Abstract:

The prevailing approach to formality transfer treats the task as a symmetric, bidirectional operation between informal and formal registers. However, we contend that this perspective masks a critical flaw in the supervision design of current benchmarks, such as GYAFC. In these datasets, binary human rewrites capture relative stylistic changes rather than absolute, human-centric definitions of formality. As a result, models are incentivized to produce "pseudo-formal" text that merely satisfies benchmark labels, often failing to generate truly formal language.

To measure the impact of this misalignment, we re-evaluated benchmark formal labels against a human-aligned definition of formality. Our analysis uncovered significant discrepancies that correlate with consistent failures in informal-to-formal generation across various model architectures. To rectify this, we propose reconceptualizing formality transfer not as a binary attribute, but as a graded spectrum. We introduce a three-tiered framework comprising informal, casual, and formal states, positioning "casual" as a distinct intermediate anchor to clarify supervision signals.

Leveraging this framework, we present 3LF, a new dataset offering parallel supervision across all three levels. Empirical results show that training on 3LF significantly mitigates informal-to-formal errors and enhances alignment with human judgment. For instance, GPT-4.1-nano saw its F1 score in the informal-to-formal direction surge from 0.06 to 0.88, despite 3LF being considerably smaller than the GYAFC dataset. We further prove that these improvements cannot be achieved through in-context learning alone and offer qualitative insights into errors driven by ambiguity and meaning distortion. Ultimately, our work illustrates how supervision design influences stylistic alignment and underscores the necessity of alignment-aware benchmark construction for controllable text generation.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC