Title: MOSS-Audio Technical Report

Abstract

MOSS-Audio is presented as a comprehensive audio-language model designed to interpret speech, environmental sounds, and music. The system is capable of performing audio captioning, timestamped transcription, time-aware question answering, and audio-grounded reasoning. Architecturally, MOSS-Audio integrates a specialized audio encoder, a modality adapter, and a large language model. The encoder generates temporal representations at a rate of 12.5 Hz, which the adapter then maps into the decoder’s space to facilitate autoregressive text generation.

Two core innovations define the system’s design: DeepStack cross-layer feature injection, a mechanism that allows the decoder to access acoustic data from various depths of the encoder, and time markers, which embed explicit temporal cues directly into the audio-token stream.

Regarding data preparation, we developed an event-preserving annotation pipeline. This process segments raw audio according to coherent event boundaries, applies annotation specific to speech, music, or general audio branches, and subsequently combines these into unified captions for pretraining. Additionally, intermediate branch-specific captions are preserved to facilitate the creation of task-oriented supervised fine-tuning (SFT) datasets.

The model undergoes pretraining on extensive audio-language corpora, incorporating time-aware objectives to bolster temporal grounding. This is followed by multi-stage post-training to refine instruction-following capabilities and audio-grounded reasoning. We introduce 4B and 8B parameter variants, available in both Instruct and Thinking configurations. MOSS-Audio demonstrates robust performance in general audio understanding, speech captioning, automatic speech recognition (ASR), and timestamped ASR, establishing it as a strong foundational model for the development of future voice agents.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC