Title: The Hidden Influence of Batch Size: Addressing Hyperparameter Bias in LoRA Assessments

Abstract: While low-rank adaptation (LoRA) has become a conventional method for fine-tuning large language models, its various iterations frequently cite conflicting empirical improvements, even when evaluated on identical benchmarks. This study identifies a previously neglected variable responsible for these discrepancies: batch size. We demonstrate that when the batch size is appropriately optimized, standard LoRA can achieve performance levels comparable to those of more intricate variants. Additionally, we introduce a cost-effective, proxy-driven approach for optimizing batch size, which illuminates how rank, dataset volume, and model capacity influence the ideal setting. These results reposition batch size from a trivial implementation choice to a critical design parameter, thereby resolving earlier contradictions and facilitating more robust comparisons of LoRA modifications.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC