Title: PolarMem: A Training-Free Polarized Latent Graph Memory for Verifiable Vision-Language Models

Abstract:

Memory serves a purpose beyond simple data storage for intelligent systems; it acts as a framework for organizing evidence and regulating belief. This function is particularly critical in multimodal reasoning, where retrieved evidence must align with the query while maintaining visual consistency. Yet, existing memory architectures for vision-language models (VLMs) are predominantly positive-associative. They tend to recall information that is similar or has been encountered before, but they do not explicitly retain data that has been verified as absent or logically ruled out.

To address this limitation, we introduce PolarMem, a training-free polarized latent graph memory framework designed for verifiable vision-language reasoning. PolarMem converts frozen VLM perceptual signals into three distinct memory states—HAS, NOT_HAS, and Uncertain—by employing semantic consistency verification and adaptive distributional partitioning. These states are then archived within a polarized graph that features separate positive and negative memory relations.

During the inference phase, our approach utilizes a lexicographical logic-aware retrieval protocol. This mechanism prioritizes logical consistency over semantic similarity, effectively filtering out conflicting memories before they are incorporated into the model’s context. We evaluated PolarMem across eight frozen VLM backbones and six multimodal benchmarks. The results demonstrate that PolarMem consistently enhances performance on retrieval-intensive tasks and minimizes contradictions at the retrieval level. These findings underscore the importance of negative memory as a vital component for developing more dependable multimodal memory systems. The code for this project is available at https://github.com/czs-ict/PolarMem.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC