Title: Enhancing Driver Monitoring with Vision-Language Models: Introducing the Drive&Act Description Dataset

Abstract

Accurately interpreting nuanced driver behaviors is a critical requirement for developing robust driver monitoring systems. However, current vision-language models (VLMs), which are typically trained on general-purpose datasets, often fail to distinguish subtle variations in driver conduct. To overcome this challenge, this study introduces a comprehensive natural language extension of the Drive&Act dataset. We assessed the capabilities of three distinct VLMs on this novel benchmark by employing LLM-based evaluation metrics. The results indicate that these models are currently unable to consistently produce precise, fine-grained descriptions of driver activities.

Leveraging the annotated Drive&Act data, we constructed a new Drive&Act description dataset featuring detailed annotations designed to train VLMs specifically for driver activity comprehension. In cross-dataset evaluations using the Driver Monitoring Dataset (DMD), the VLM fine-tuned on our newly created description dataset demonstrated strong generalization capabilities when applied to actions within the DMD. Specifically, the model fine-tuned on the Drive&Act description dataset attained an ACCR score of 76, significantly surpassing the zero-shot VLM baseline, which achieved an ACCR of 66.

These outcomes underscore that equipping VLMs with richly annotated driver action descriptions substantially enhances their capacity to interpret driver behavior. Furthermore, the study points to the necessity of incorporating more diverse datasets to facilitate broader generalization in future applications. The Drive&Act description dataset and the associated code will be made publicly accessible via GitHub.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC