Title: SketchSong: Hierarchical Song Generation with Sketch Planning and Fine-Grained Multi-Track Modeling

Abstract:

While contemporary song generation systems are capable of producing realistic audio, creating full-length, coherent songs remains a significant challenge due to two primary limitations. First, existing methods often lack explicit song-level arrangement planning. Consequently, models are forced to simultaneously manage high-level structural development and low-level audio synthesis, which frequently results in structural incoherence, characterized by abrupt section transitions and a lack of dynamic progression. Second, the coarse representation of individual musical parts tends to blur their distinct functions and interactions, thereby restricting the richness of the final arrangement.

To resolve these issues, we introduce SketchSong, a hierarchical framework for song generation that leverages song-level sketch planning and fine-grained multi-track modeling. In terms of temporal structure, SketchSong operates through a coarse-to-fine approach: it first predicts a concise sequence of high-level sketch tokens extracted from compressed audio representations, and subsequently generates detailed audio tokens conditioned on these sketches. This process ensures that an explicit arrangement plan is established prior to the generation of intricate audio details. Regarding the track dimension, the framework explicitly models four distinct components: vocals, bass, drums, and other instruments. This architecture allows the model to more accurately capture the specific roles and interplay among different musical elements.

Evaluations on song generation benchmarks demonstrate that SketchSong consistently surpasses our baseline across both objective metrics and human listening tests. Notably, even without utilizing additional post-training techniques for preference optimization—such as aligning lyrics or text prompts—SketchSong delivers results competitive with strong, post-trained open-source systems, underscoring the efficacy of our proposed design.

Source: arXiv Generated at: 2026-06-03 00:00:00 UTC