Title: ProjQ: Project-and-Quantize for Adapter-Aware LLM Compression

Abstract:

The standard approach for deploying Large Language Models (LLMs) efficiently typically combines Post-Training Quantization (PTQ) and Low-Rank Adaptation (LoRA). However, implementing these techniques in sequence creates a significant bottleneck: PTQ tends to distribute random noise across the model’s weights in a manner that LoRA cannot readily correct. Consequently, LoRA expends its limited capacity attempting to rectify this uncorrectable noise, rather than focusing on enhancing task-specific performance.

To address this issue, we introduce ProjQ, a new framework that restricts quantization noise to the low-rank manifold through orthogonal subspace projection. We developed an efficient alternating algorithm designed to structure the quantization noise as low-rank, thereby transferring dominant error components to the subsequent adapter. This process minimizes the residual error within the orthogonal "uncorrectable" subspace. Our theoretical findings indicate that ProjQ maintains significantly higher model plasticity for downstream applications than conventional PTQ methods.

Extensive testing on LLaMA-2, Qwen2.5, and Qwen3 demonstrates that ProjQ consistently surpasses current methods in both compensating for quantization errors and fine-tuning for downstream tasks. The method achieves up to $2\times$ lower evaluation loss for error compensation and matches the performance of standard 4-bit baselines on language modeling tasks while utilizing only 3 bits. The code is available on https://github.com/yy9301/ProjQ .

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC