Title: A Comparative Study of Multimodal Strategies for Classifying Visually Complex Documents

Abstract:

Classifying document types within visually dense materials presents a significant challenge, as critical data is scattered across textual, visual, and structural dimensions. To manage this intricacy, existing methods employ a wide array of multimodal modeling techniques, leading to fragmented architectures that hinder straightforward systematic evaluation. This architectural diversity is mirrored in prior comparative studies, which frequently utilize inconsistent experimental setups. Such inconsistencies obscure meaningful comparisons and make it difficult to gauge advancements in the field.

To overcome these obstacles, this study offers a structured examination of multimodal design patterns in both transformer-based and large language model (LLM) architectures, supported by a controlled empirical assessment within a standardized experimental environment. We evaluate four distinct models—LayoutLMv3, Donut, Qwen3-VL-32B-Instruct, and Qwen3-32B—using the RVL-CDIP benchmark. This approach allows for a systematic dissection of how text, image, and layout inputs contribute to classification accuracy, with a specific emphasis on differentiating between OCR-dependent and OCR-free methodologies.

Our findings indicate that specialized multimodal transformers surpass LLM-based solutions when handling documents characterized by heavy visual content and complex layouts. While information extracted via OCR offers valuable supplementary support, visual data proves to be the most potent factor in ensuring classification reliability. These results underscore the continued necessity of multimodal processing for documents with distinct structural arrangements. Ultimately, this research establishes a rigorous foundation for comparing multimodal systems and delivers actionable insights for choosing optimal feature sets and architectural designs for document type classification.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC