Title: E2LLM: Advancing Efficient LLM Serving in Heterogeneous Edge and Fog Networks

Abstract:

While Large Language Models (LLMs) are now central to contemporary applications, their practical implementation faces significant hurdles. Effective deployment requires not only model execution but also careful management of cost efficiency, latency reduction, and resource optimization. Standard methodologies often presuppose that a complete model can reside on a single device, an assumption that frequently fails in real-world contexts, especially within Edge and Fog ecosystems where hardware capabilities are limited.

To address these constraints, this study presents E2LLM, a novel framework engineered for efficient LLM deployment in resource-scarce environments. Instead of merely splitting one model across all available nodes, E2LLM employs a strategy of replicating the entire model across several device groups, known as replicas. Within each replica, model parallelism is applied. Furthermore, each replica is designated a specific function—either PREFILL or DECODER—determined by its proficiency in processing input versus output tokens. This division capitalizes on the distinct characteristics inherent to the two phases of LLM inference.

To optimize device organization, the framework employs a Genetic Algorithm to create clusters that maximize overall system performance. Subsequently, Dynamic Programming is utilized within each cluster to identify the optimal partitioning strategy, thereby minimizing execution bottlenecks during model-parallel operations. Our experimental findings indicate that E2LLM demonstrates strong adaptability to fluctuating workloads, including cases with substantial disparities in input and output token lengths. Notably, under high-demand scenarios, E2LLM achieves a reduction in average waiting time of more than 50% when compared to the Splitwise baseline.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC