Title: The Gold Standard Isn’t Always Standard: Navigating the Complexities of Evaluating User-Generated Content Translation

Abstract:

The translation of user-generated content (UGC) presents unique evaluation hurdles due to its reliance on non-standard linguistic features, ranging from typographical errors to expressive elements like slang, character repetition, and emojis. Determining the quality of a UGC translation is complex, as the definition of a "good" translation is contingent upon the intended level of standardization in the final output. To investigate this issue, we analyzed the human translation guidelines associated with four distinct UGC datasets. From this analysis, we constructed a taxonomy comprising twelve categories of non-standard phenomena and five corresponding translation actions: NORMALISE, COPY, TRANSFER, OMIT, and CENSOR. Our findings highlight significant variations in how UGC is handled across datasets, creating a wide spectrum of standardness within reference translations. Furthermore, we demonstrate that large language models exhibit high sensitivity to prompts containing explicit instructions for UGC translation, achieving better performance when their outputs align with specific dataset guidelines. We contend that equitable evaluation necessitates that both models and metrics remain cognizant of these translation guidelines. Ultimately, we advocate for the establishment of clearer guidelines during the dataset creation process and the development of evaluation frameworks that are both controllable and aware of specific guidelines for UGC translation.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC