Title: Beyond Semantics: Modeling Factual and Affective Perceptual Experiences from Vision-Language Data

Abstract:

This paper introduces P-Topics (Perception Topics) modeling, a novel framework designed to investigate how images are perceived through both affective and cultural lenses. Our primary objectives are twofold: first, to identify and model distinct perception experiences within datasets comprising images and captions, where each experience is characterized by an objective factual component and a subjective affective dimension; and second, to map images to their corresponding perception experiences. To address the challenges of P-Topics modeling, we propose PercepT (Perception topic Transformer), a two-stage architectural approach. During the formation phase, PercepT identifies P-Topics as visual-textual clusters through an unsupervised training objective, dynamically determining the optimal number of clusters to align with the dataset’s perceptual complexity. In the subsequent mapping phase, the model employs attention pooling to learn P-Topic mapping functions, thereby linking images to their respective clusters.

Experimental results on the ArtELingo dataset demonstrate PercepT’s superiority. It achieves a silhouette score of 0.97, significantly outperforming the closest baseline, which scored 0.37, indicating the creation of more coherent perceptual clusters. Furthermore, PercepT attains an AUC score of 0.94, compared to the baseline’s 0.77, highlighting its enhanced ability to map images to perceptual clusters. Human evaluations further validate that PercepT captures semantically rich perception experiences, substantially surpassing existing methods. We will release the source code for public access.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC