Title: LoRA as Knowledge Memory: An Empirical Investigation

Abstract:

While the need for continuous knowledge updates in pre-trained large language models (LLMs) is growing, achieving this efficiently remains a significant hurdle. Popular inference-time strategies such as In-Context Learning (ICL) and Retrieval-Augmented Generation (RAG) are widely used, yet they are often limited by context window constraints, high costs, and issues related to retrieval fragmentation. Moving away from these context-dependent frameworks, this study explores a parametric alternative: utilizing Low-Rank Adaptation (LoRA) as a modular knowledge memory. Despite recent interest in this concept, the underlying mechanisms that dictate its capacity and composability have not been thoroughly investigated. To address this gap, we present the first comprehensive empirical analysis mapping the design space of LoRA-based memory. Our work covers a broad spectrum of topics, including characterizing storage capabilities, optimizing internalization processes, scaling multi-module architectures, and assessing performance in long-context reasoning tasks. Instead of advocating for a specific architectural design, we offer practical insights into the operational limits of LoRA memory. Our results suggest that LoRA serves as a complementary memory axis alongside RAG and ICL, providing unique benefits. The associated code and datasets can be accessed at https://github.com/ahn-ml/Understanding-LoRA-as-Knowledge-Memory.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC