ES4R:
Speech Encoding Based on Prepositive Affective Modeling for Empathetic Response Generation

School of Computer Science and Engineering, Northeastern University, Shenyang 110819, China
Annual Meeting of the Association for Computational Linguistics (ACL) 2026

*Corresponding Author

Abstract

Empathetic speech dialogue requires not only understanding linguistic content but also perceiving rich paralinguistic information such as prosody, tone, and emotional intensity for affective understandings. Existing speech-tospeech large language models either rely on ASR transcription or use encoders to extract latent representations, often weakening affective information and contextual coherence in multiturn dialogues. To address this, we propose ES4R, a framework for speech-based empathetic response generation. Our core innovation lies in explicitly modeling structured affective context before speech encoding, rather than relying on implicit learning by the encoder or explicit emotion supervision. Specifically, we introduce a dual-level attention mechanism to capture turn-level affective states and dialogue-level affective dynamics. The resulting affective representations are then integrated with textual semantics through speechguided cross-modal attention to generate empathetic responses. For speech output, we employ energy-based strategy selection and style fusion to achieve empathetic speech synthesis. ES4R consistently outperforms strong baselines in both automatic and human evaluations and remains robust across different Large Language Model (LLM) backbones.

Architecture

ES4R Architecture The overall framework of ES4R. Stage 1: Intra-Turn and Inter-Turn attention are used to prepositive affective modeling. Stage 2: Speech-guided cross-modal fusion is used for empathetic response generation. Stage 3: Energy-based strategy selection and style fusion are applied for speech synthesis.

Analyses

Demo Examples

Example 1
Turn Audio Input Transcript Audio Output Text Output
T1 I've been struggling to balance work and family responsibilities, and it's really taking a toll on me. That sounds really tough. It must be exhausting trying to manage everything at once.
T2
Response
Yes, it feels like I'm constantly running on empty and never getting a break. I can imagine how that feels. It's important to take care of yourself and prioritize your well-being, even if it means saying no to some things or asking for help when you need it.
Example 2
Turn Audio Input Transcript Audio Output Text Output
T1 I just can't shake off this feeling of being stuck in my life. It's like every day is the same and I can't see a way out. It sounds really tough to feel that way. It's understandable to feel overwhelmed when life feels repetitive.
T2 Sometimes, I wonder if I’m just not trying hard enough to change things. It's common to feel that way, but it doesn’t mean you’re not trying. Change can be slow and requires patience.
T3
Response
I guess I just need to remind myself that small steps can lead to progress. That's a great reminder. Sometimes it can be hard to see progress when things feel slow, but every small step counts towards making changes.
Example 3
Turn Audio Input Transcript Audio Output Text Output
T1 I'm really struggling to keep up with everything lately. It sounds like you're carrying a heavy load. Do you want to talk about what's been on your mind?
T2 It's like I can't find a moment to breathe. Work is demanding and my family needs me too. That sounds exhausting. It's tough when you feel like there's no break in sight.
T3
Response
I just want to feel like I'm managing things instead of feeling lost. I understand how you feel. It can be overwhelming to try and balance everything, but remember that it's okay to ask for help when you need it. Maybe talking to someone about your stress could make a difference.