Title: Parameter-Efficient Fine-Tuning with Learnable Rank

Abstract:

Low-Rank Adaptation (LoRA) has emerged as a widely adopted parameter-efficient fine-tuning (PEFT) technique. It limits weight modifications to low-rank adapters, thereby imposing a static low-rank inductive bias by conducting optimization within a low-dimensional subspace. This study challenges the assumption that a rigid rank constraint constitutes the most optimal inductive bias for PEFT. To address this, we propose Learnable Rank LoRA (LR-LoRA), a novel PEFT approach where the adapter rank is dynamically learned throughout the training phase. Unlike methods that enforce a uniform rank across all adapter layers, LR-LoRA empowers the optimizer to select the ideal rank for each individual layer. Our findings reveal significant layer-specific differences in the learned ranks, with the attention and multi-layer perceptron (MLP) components of transformer models showing distinct rank preferences. Evaluated across various benchmarks for language comprehension and commonsense reasoning, LR-LoRA secures state-of-the-art results in the majority of scenarios and consistently surpasses robust PEFT baselines. These results underscore that employing a learnable rank offers a more adaptable and potent inductive bias compared to fixed-rank strategies.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC