Title: OP-LoRA: The Blessing of Dimensionality

Abstract:

Low-rank adapters (LoRA) facilitate the fine-tuning of large models by introducing only a minimal set of parameters. Despite this efficiency, they frequently encounter optimization difficulties due to an ill-conditioned loss landscape. Previous solutions have attempted to mitigate these issues by aligning adapter updates with full fine-tuning gradients through specialized optimizers; however, such approaches are often computationally intensive and lack the flexibility to support emerging adapter designs.

To address these limitations, we present OP-LoRA, a novel technique that temporarily substitutes standard LoRA adapters with weights generated by an auxiliary Multi-Layer Perceptron (MLP). This MLP is removed following the training process, meaning that while additional parameters are utilized during optimization to enhance convergence, there is no inference overhead. This method not only reduces wall-clock time compared to custom optimizers but also incurs zero extra cost at inference. Furthermore, adapting OP-LoRA to new adapter architectures is straightforward, requiring only adjustments to the prediction head size for each specific adapter type.

Our results demonstrate that OP-LoRA enables the optimization process to dynamically adjust step sizes, thereby boosting performance and reducing sensitivity to learning rate selection. Across both small- and large-scale LoRA tuning tasks, OP-LoRA consistently outperforms standard LoRA and its variants. Notably, in image generation tasks, OP-LoRA achieved CMMD score improvements of up to 15 points over LoRA, effectively matching LoRA’s performance while utilizing half the number of inference parameters.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC