Aqoonsiga Dareenka Hadalka

Speech emotion recognition is a classification or regression task: a system receives a stretch of recorded speech and outputs an estimate of the emotion it expresses. The output is usually either a category, such as neutral, happy, sad or angry, or scores on emotional dimensions. The three dimensions used in most research are valence (how positive or negative), arousal or activation (how calm or excited) and dominance (how in control). The system works from how something is said, including pitch, loudness, timing and voice quality, and in many modern systems also from the words. It estimates expressed or perceived emotion, what listeners would judge, not the speaker's inner feelings.

A typical pipeline has four steps. First, the audio is cleaned and cut into utterances, often a few seconds long. Second, features are computed. Classic systems used hand-designed measurements; the eGeMAPS set, extracted with the openSMILE toolkit, includes pitch, loudness, spectral shape and voice-quality measures taken every 10 milliseconds. Modern systems instead pass the waveform through a self-supervised speech model such as wav2vec 2.0, a network pretrained on large amounts of unlabeled speech, and use its internal layers as features. Third, a small classifier or regressor, trained on labeled examples, maps features to emotions. Fourth, frame-level predictions are pooled into one decision per utterance.

The switch to pretrained representations produced large gains. In a 2021 study, Leonardo Pepino, Pablo Riera and Luciana Ferrer compared feature types using the same downstream model. On IEMOCAP, eGeMAPS features reached 52.4% average recall, meaning recall averaged across emotion classes so rare classes count equally, while a weighted combination of all wav2vec 2.0 layers reached 67.2%. On RAVDESS the same comparison went from 57.0% to 84.3%. They also found that the version of wav2vec 2.0 fine-tuned for speech recognition did worse than the purely pretrained one, apparently because fine-tuning for transcription discards tone information. Emotion-specific pretraining followed: emotion2vec, posted in December 2023 by Ziyang Ma and colleagues, was pretrained on unlabeled emotional speech and outperformed general speech models on IEMOCAP using only linear layers.

Datasets shaped the field. IEMOCAP, built at the University of Southern California, holds about 12 hours of audiovisual recordings of actors in dyadic, meaning two-person, sessions, both scripted and improvised, annotated by multiple raters with categories and dimensional labels. RAVDESS, published in PLOS ONE in 2018, contains 7,356 recordings by 24 professional actors, 12 female and 12 male, in eight emotions. Both are acted. Their weakness is visible in RAVDESS's own validation: human raters identified the intended emotion about 60% of the time from audio alone, against about 80% with audio and video. The MSP-Podcast corpus, described in a 2025 paper by Carlos Busso and colleagues, moved to natural speech: over 400 hours from podcasts shared under licenses that allow redistribution, each sample rated by at least five people.

The gap between acted and natural speech is the central limitation. The Interspeech 2025 Challenge on Speech Emotion Recognition in Naturalistic Conditions, organized by Busso's group and collaborators, used speaker-independent splits so models were tested on unseen people. Organizers reported a record number of participants and found that the best systems combined audio and text foundation models. Even so, one published entry reported a macro F1 score, the average of per-class F1 scores, of 39.79% on the test set, far below the scores common on acted corpora. Natural emotions are subtle, mixed and unevenly distributed, with neutral speech dominating.

Compared with alternatives, voice has strengths and weaknesses. Text sentiment analysis reads the words and misses sarcasm or tone; voice captures tone but is confused by accents, microphones and background noise. Facial expression analysis needs a camera and faces the same question of what outward signals reveal. Multimodal systems that combine voice and text currently perform best on benchmarks. Physiological signals such as heart rate are harder to fake but need sensors. None measures felt emotion directly; all infer from outward signals.

Labels are inherently uncertain. Raters often disagree on the same clip, which is why corpora collect several ratings and why MSP-Podcast records both a primary emotion and secondary perceived emotions. Expression varies by culture, language, age and individual, so a model trained on one population can misread another. Acted datasets exaggerate emotions. Accuracy reported on one corpus often drops on another. These issues raise fairness concerns when results affect people, and they underlie the EU AI Act's prohibition, applying from February 2, 2025, on AI systems that infer emotions of people in workplaces and educational institutions, with exceptions for medical or safety reasons.

In practice, building an SER system requires labeled emotional speech that matches the target setting, which is the scarce resource. Freely available corpora carry license limits; RAVDESS, for example, is released under a non-commercial Creative Commons license. Compute needs are modest: a pretrained speech model can be frozen and only a small classifier trained, as in the emotion2vec and wav2vec 2.0 studies. The real costs are collecting consented, representative data, multiple human ratings per clip, evaluation on unseen speakers and a legal review of where the system may be used.