Title: Accelerating and Enhancing Multi-Byte Prediction via Probabilistic Circuits

Abstract:

Multi-token prediction (MTP) has emerged as a key technique for drastically reducing generation times in large language models (LLMs), particularly benefiting byte-level LLMs. Although these models eliminate the need for tokenizers, they typically suffer from significant computational slowness. Current MTP approaches often face a dilemma: they either rely on independence assumptions regarding future tokens, which limits their expressive power, or they produce tokens sequentially within a specified window, which adds to latency. This study examines the balance between expressiveness and speed in MTP through the lens of probabilistic circuits (PCs).

We introduce MTPC, a framework that enables the exploration of various methods for encoding joint distributions over upcoming tokens by choosing specific circuit architectures. This approach generalizes several classical models, including hidden Markov models, tensor networks, and hierarchical mixture models. To demonstrate the practical value of MTPC, we applied it to existing byte-level LLMs like EvaByte and byte-adapted subword models such as Llama3.2 3B.

Our experimental results indicate that when integrated with speculative decoding, MTPC offers a substantial increase in generation speed compared to MTP methods based on independence assumptions, all while preserving the performance standards of the original verifier LLM. Furthermore, we conduct a thorough analysis of the optimal compromise between expressiveness and latency by investigating various parameterizations of MTPC, including different PC architectures and the extent of partial layer sharing between the verifier and draft LLMs.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC