Title: DeInfer: Streamlining Parallel Inference for Decomposed Large Language Models

Abstract: While current research on large language model (LLM) decomposition primarily targets enhanced performance on downstream tasks, it frequently overlooks the significant bottlenecks in parallel inference performance that arise as model sizes increase. To address this critical efficiency gap, we present DeInfer, a specialized high-performance inference framework designed explicitly for the parallel processing of decomposed LLMs. The system integrates a suite of optimizations aimed at maximizing throughput while maintaining compatibility with state-of-the-art optimization methods. Comprehensive experimental evaluations underscore DeInfer’s superior performance, indicating its potential to substantially advance the parallel inference capabilities of decomposed LLMs.

Source: arXiv Generated at: 2026-06-04 00:00:00 UTC