Title: Investigating Low-Rank PEFT with Finer Parameter Steps: A Controlled Analysis Using CP Tensor Adapters

Abstract:

Evaluations of low-rank adapters typically involve testing a limited range of ranks; however, the chosen rank also dictates the granularity of the available parameter budget. For instance, in a $2048{\times}2048$ OPT attention projection, incrementing LoRA by a single rank adds $4096$ trainable scalars, creating significant discontinuities between viable low-budget adapter configurations. This study investigates whether employing a tensorized adapter structure with more granular capacity increments alters the relationship between accuracy and budget. We explore this hypothesis using canonical polyadic (CP) tensor adapters with fixed components. In a $32{\times}64{\times}32{\times}64$ tensorization setup, each normalized CP component introduces only $193$ trainable scalars per projection—approximately $21$ times fewer than the step size of a single LoRA rank. We conducted a comparative analysis of CP adapters and LoRA on the OPT-1.3B model across the SST-2, RTE, and BoolQ datasets, ensuring consistency in target modules, training protocols, data limits, and random seed schedules. The results indicate that while CP adapters offer stable training and bridge the gaps between LoRA rank sizes, their impact varies by task. SST-2 achieves a low-budget performance plateau early on, BoolQ improves with additional CP components before slightly underperforming compared to LoRA, and RTE continues to favor LoRA. Consequently, while finer parameter steps are valuable for diagnosing sensitivity to PEFT budgets, they do not inherently ensure an improved accuracy-to-budget trade-off.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC