Title: Social Caption: Evaluating Social Understanding in Multimodal Models

Abstract: For multimodal large language models (MLLMs), the capacity to interpret human social interactions is vital. This paper presents SOCIAL CAPTION, an evaluation framework rooted in interaction theory designed to assess MLLMs across three distinct dimensions: Social Inference (SI), which measures the precision of interaction-related deductions; Holistic Social Analysis (HSA), which gauges the generation of comprehensive interaction descriptions; and Directed Social Analysis (DSA), which assesses the extraction of pertinent information from social contexts. The study investigates key factors affecting performance in social understanding, including model scale, architectural choices, and the presence of spoken context. Furthermore, experiments involving MLLM-based judges highlight a viable trajectory for scaling the automated assessment of multimodal social comprehension.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC