Title: EntangleCodec: A Unified Discrete Audio Tokenizer via Semantic-Acoustic Entanglement

Abstract:

Audio tokenizers function as the discrete bridge connecting continuous audio signals with Audio Language Models (ALMs). However, current solutions frequently face difficulties in simultaneously supporting both generation and understanding tasks. Codecs designed primarily for reconstruction maintain high acoustic fidelity but often lack deep semantic content. Conversely, tokenizers that are aware of semantics typically utilize distinct streams for semantic and acoustic data, which can lead to redundancy or misalignment issues.

To address these challenges, we introduce EntangleCodec, a unified discrete audio tokenizer that acquires semantic-acoustic representations aligned with captions prior to quantization. By matching audio against detailed captions instead of standard ASR transcripts, EntangleCodec efficiently encodes linguistic details, speaker identity, prosody, emotion, and acoustic environments into a compact token stream. This architecture is complemented by a flow-matching diffusion decoder, which facilitates high-quality reconstruction for speech, music, and general audio.

EntangleCodec delivers reconstruction performance comparable to specialized codecs while surpassing all codec-based baselines in audio understanding tasks, achieving a performance gain of up to +7.4% on the MMAR benchmark. Additionally, it enables a unified framework for both Text-to-Speech (TTS) and Text-to-Audio (TTA) generation.

Our experiments reveal strong scaling properties for audio language models built on EntangleCodec. At a modest size of 0.6B parameters, the model outperforms specialized continuous-representation LLMs with over 13B parameters across three benchmarks, utilizing 22$\times$ fewer parameters. Scaling the model up to 8B parameters establishes new state-of-the-art results on MMAR, underscoring that the quality of representations is just as vital as model scale in audio language modeling.

Code and model weights are publicly accessible at https://github.com/luckyerr/EntangleCodec.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC