Title: Urban Perception Benchmarks for Vision-Language Models Must Prioritize Reliability and Negotiation

Abstract:

As Vision-Language Models (VLMs) become more prevalent in generating structured descriptions of street-level imagery, their applications now span critical civic functions such as public consultation, mapping, and streetscape auditing. These tasks inherently blend observable physical attributes with subjective appraisal categories. Consequently, human reference data often reflects not just consensus, but also distributions of judgment, including instances of disagreement and explicit non-response.

This study contends that evaluating VLMs for urban perception requires a paradigm shift: disagreement and abstention should be recognized as valid measurement outcomes. Furthermore, benchmarking protocols must report inter-annotator reliability alongside model alignment. Crucially, when model outputs are intended to guide urban governance, the label space and scoring policies should be treated as negotiable artifacts rather than fixed truths.

To substantiate this argument, we present a benchmark involving 100 street scenes in Montreal. These scenes were annotated across 30 distinct dimensions by 12 participants representing seven different community organizations. We also conducted a deterministic zero-shot evaluation of seven VLMs. Our findings indicate that, across various dimensions, the degree of agreement between models and human consensus correlates strongly with the reliability of human judgments at the dimension level. However, in the "Overall Impression" appraisal dimension, both models and human annotators displayed a distributional mismatch, particularly regarding differing rates of "Not applicable" responses.

We conclude by outlining specific actions for benchmark creators, model developers, and institutional stakeholders. These steps aim to ensure that uncertainty and the underlying assumptions of benchmarks are made transparent in evaluation reports.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC