Title: Enhancing Music Recommendations with LLMs via a Multimodal Approach

Abstract

Traditional music recommendation engines often treat tracks as indistinguishable tokens, depending heavily on collaborative filtering and user interaction histories. This approach tends to ignore the intrinsic semantic and acoustic properties of the music. While previous studies have investigated sequential recommendation methods augmented by Large Language Models (LLMs), multimodal data, or textual enhancements, existing solutions only partially integrate semantic, acoustic, or engagement metrics. None have successfully modeled all three dimensions together within a cohesive LLM-based sequential reasoning system that anchors suggestions in the actual content of the songs.

To address this gap, we introduce a multimodal framework designed for session-based music recommendation. We enhance the LastFM-1K dataset by incorporating three distinct signals: (1) embeddings for audio and lyrics derived from pretrained music and text representation models; (2) semantic metadata generated by LLMs, formatted according to the MGPHot annotation schema; and (3) metrics reflecting listening completion ratios.

Our methodology builds upon the E4SRec framework, adapting it to accommodate multimodal features and various item ID encoder backbones, such as SASRec, BERT4Rec, and GRU4Rec. Additionally, we expand the LLM backbone capabilities by testing LLaMa-2-13B, Qwen2.5-7B-Instruct, and LLaMa-3-70B, evaluating performance in both zero-shot and fine-tuned configurations.

Experimental results demonstrate that incorporating content-based features significantly outperforms ID-only baselines, achieving improvements of up to 95% in Recall and 79% in NDCG. However, our findings also reveal that straightforward multimodal fusion does not consistently produce additive benefits, underscoring the complexities involved in cross-modal integration. To facilitate further research, we are releasing a comprehensive, large-scale multimodal benchmark for music recommendation.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC