Title: HybridThinker: Streamlining Chain-of-Thought Reasoning with Compressed Memory and Ephemeral Thought Steps

Abstract: Although extended chain-of-thought (CoT) traces significantly enhance the reasoning capabilities of Large Language Models (LLMs), they come with high computational and memory overhead. Current approaches to CoT compression attempt to address this by condensing thought steps into compact memory tokens, retaining only these summaries during inference. However, this process often discards fine-grained details, leading to increased errors in subsequent reasoning steps. To address this limitation, we introduce HybridThinker, a method that preserves these compact representations while also temporarily retaining the original thought steps to supply necessary granular information.

We identified a critical issue during training: if thought steps are fully accessible to later steps, the model tends to bypass the memory tokens entirely, retrieving information directly from the transient steps. This behavior prevents the model from adequately learning how to compress and retrieve data via memory tokens. To resolve this, we developed a hybrid training protocol. In this scheme, only a subset of thought steps is made directly accessible via attention mechanisms; the remaining steps are masked, compelling the model to rely on memory tokens for compression and retrieval.

Evaluated across four reasoning benchmarks, HybridThinker performs on par with uncompressed baselines. It sets a new state-of-the-art for CoT compression, achieving an average accuracy improvement of 5.8 points compared to existing methods, all while maintaining comparable inference speeds. Ablation studies further validate that both the retention of temporary thought steps and the implementation of the hybrid training scheme are essential contributors to these performance gains.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC