Title: NextMotionQA: Evaluating Human Motion Comprehension via Vision-Language Models

Abstract:

Accurate assessment of human motion understanding is a cornerstone for progress in animation, robotics, and embodied AI. Yet, current benchmarks are hindered by imprecise semantic granularity, uniform difficulty levels, subpar annotation quality, and widespread answer ambiguity, rendering them ineffective at pinpointing specific model failures. To address these shortcomings, we present NextMotionQA, a robust benchmark that utilizes vision-language models (VLMs) to create a semi-automated dataset verified by experts.

NextMotionQA comprises three distinct tasks: fine-grained error correction, video captioning, and multiple-choice question answering. These tasks are organized along three fundamental semantic axes and divided into three tiers of complexity. Through an extensive evaluation of twelve prominent VLMs, we identify significant capability gaps and weaknesses that conventional, single-task assessments typically overlook.

In a parallel investigation, we examine the efficacy of VLMs as judges for text-to-motion generation—a practice gaining traction in recent research. We analyze whether these models exhibit performance declines when faced with more challenging tasks. Our findings indicate that while VLMs correlate well with expert ratings on broad criteria (Cohen's $\kappa=0.70$), their reliability collapses when required to make fine-grained, part-level judgments (Cohen's $\kappa=0.10$). This result confirms the utility of the VLM-as-judge paradigm in high-confidence scenarios while clearly delineating its limitations.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC