Title: Sympatheia: A Voice Assistant Capable of Emotional Adaptation via Continuous Affect Conditioning

Abstract:

To generate appropriate responses, empathetic spoken dialogue systems need to accurately infer a user’s emotional state. However, daily speech frequently presents challenges due to weak, neutral, or ambiguous emotional signals. In response to this challenge, we present Sympatheia, a speech-to-speech dialogue framework. This system is conditioned on affect derived from the user’s voice and, when accessible, incorporates explicit affect specifications. These specifications are delivered as a continuous valence-arousal (VA) control signal, sourced either from a user interface or a multimodal sensing module.

For model training, we developed Sympatheia-18k, a synthetic spoken dialogue corpus designed for emotion conditioning. This dataset features 12 distinct emotion anchors and is divided into two specific splits: an emotional split dedicated to learning affective speech behaviors, and a neutral split. The neutral split pairs emotionally neutral queries with multiple emotion-conditioned responses, allowing for the isolation of explicit emotion control in scenarios involving emotional ambiguity.

Our empirical findings indicate that Sympatheia surpasses speech conversational baselines. It generates responses that are not only semantically suitable but also vocally delivered in an emotionally appropriate manner. Furthermore, we demonstrate that the VA interface can consolidate emotion estimates from various sensing modules, such as facial expressions, biosignals, and textual affect descriptions. This integration enhances response alignment, particularly when speech data alone offers insufficient emotional evidence. Collectively, these results highlight that continuous affect conditioning serves as a practical and effective method for developing emotionally adaptive voice assistants.

Source: arXiv Generated at: 2026-06-02 00:00:00 UTC