概述
它對於呼叫分析、健康研究和語音助理很重要,但自然語音的準確性較低,歐盟自 2025 年 2 月起禁止在工人和學生中使用它。
重點摘要
- 語音情緒辨識估計聲音表達的情感,作為類別或效價、喚醒度和主導度分數。
- 在 2021 年的一項研究中,wav2vec 2.0 功能將 IEMOCAP 平均召回率從經典 eGeMAPS 功能的 52.4% 提高到 67.2%。
- 人類評估者大約 60% 的時間僅透過音訊就可以識別出 RAVDESS 的表演情緒,這表明即使對於人類來說,這項任務也有多困難。
- 在自然播客演講中,已發布的 Interspeech 2025 挑戰條目的宏觀 F1 得分為 39.79%,遠低於表演語料庫的結果。
- 自2025年2月2日起,歐盟人工智慧法案禁止在工作場所和學校進行人工智慧情感推斷,除非是出於醫療或安全原因。
深入探討
Speech emotion recognition is a classification or regression task: a system receives a stretch of recorded speech and outputs an estimate of the emotion it expresses. The output is usually either a category, such as neutral, happy, sad or angry, or scores on emotional dimensions. The three dimensions used in most research are valence (how positive or negative), arousal or activation (how calm or excited) and dominance (how in control). The system works from how something is said, including pitch, loudness, timing and voice quality, and in many modern systems also from the words. It estimates expressed or perceived emotion, what listeners would judge, not the speaker's inner feelings.
A typical pipeline has four steps. First, the audio is cleaned and cut into utterances, often a few seconds long. Second, features are computed. Classic systems used hand-designed measurements; the eGeMAPS set, extracted with the openSMILE toolkit, includes pitch, loudness, spectral shape and voice-quality measures taken every 10 milliseconds. Modern systems instead pass the waveform through a self-supervised speech model such as wav2vec 2.0, a network pretrained on large amounts of unlabeled speech, and use its internal layers as features. Third, a small classifier or regressor, trained on labeled examples, maps features to emotions. Fourth, frame-level predictions are pooled into one decision per utterance.
The switch to pretrained representations produced large gains. In a 2021 study, Leonardo Pepino, Pablo Riera and Luciana Ferrer compared feature types using the same downstream model. On IEMOCAP, eGeMAPS features reached 52.4% average recall, meaning recall averaged across emotion classes so rare classes count equally, while a weighted combination of all wav2vec 2.0 layers reached 67.2%. On RAVDESS the same comparison went from 57.0% to 84.3%. They also found that the version of wav2vec 2.0 fine-tuned for speech recognition did worse than the purely pretrained one, apparently because fine-tuning for transcription discards tone information. Emotion-specific pretraining followed: emotion2vec, posted in December 2023 by Ziyang Ma and colleagues, was pretrained on unlabeled emotional speech and outperformed general speech models on IEMOCAP using only linear layers.
Datasets shaped the field. IEMOCAP, built at the University of Southern California, holds about 12 hours of audiovisual recordings of actors in dyadic, meaning two-person, sessions, both scripted and improvised, annotated by multiple raters with categories and dimensional labels. RAVDESS, published in PLOS ONE in 2018, contains 7,356 recordings by 24 professional actors, 12 female and 12 male, in eight emotions. Both are acted. Their weakness is visible in RAVDESS's own validation: human raters identified the intended emotion about 60% of the time from audio alone, against about 80% with audio and video. The MSP-Podcast corpus, described in a 2025 paper by Carlos Busso and colleagues, moved to natural speech: over 400 hours from podcasts shared under licenses that allow redistribution, each sample rated by at least five people.
The gap between acted and natural speech is the central limitation. The Interspeech 2025 Challenge on Speech Emotion Recognition in Naturalistic Conditions, organized by Busso's group and collaborators, used speaker-independent splits so models were tested on unseen people. Organizers reported a record number of participants and found that the best systems combined audio and text foundation models. Even so, one published entry reported a macro F1 score, the average of per-class F1 scores, of 39.79% on the test set, far below the scores common on acted corpora. Natural emotions are subtle, mixed and unevenly distributed, with neutral speech dominating.
Compared with alternatives, voice has strengths and weaknesses. Text sentiment analysis reads the words and misses sarcasm or tone; voice captures tone but is confused by accents, microphones and background noise. Facial expression analysis needs a camera and faces the same question of what outward signals reveal. Multimodal systems that combine voice and text currently perform best on benchmarks. Physiological signals such as heart rate are harder to fake but need sensors. None measures felt emotion directly; all infer from outward signals.
Labels are inherently uncertain. Raters often disagree on the same clip, which is why corpora collect several ratings and why MSP-Podcast records both a primary emotion and secondary perceived emotions. Expression varies by culture, language, age and individual, so a model trained on one population can misread another. Acted datasets exaggerate emotions. Accuracy reported on one corpus often drops on another. These issues raise fairness concerns when results affect people, and they underlie the EU AI Act's prohibition, applying from February 2, 2025, on AI systems that infer emotions of people in workplaces and educational institutions, with exceptions for medical or safety reasons.
In practice, building an SER system requires labeled emotional speech that matches the target setting, which is the scarce resource. Freely available corpora carry license limits; RAVDESS, for example, is released under a non-commercial Creative Commons license. Compute needs are modest: a pretrained speech model can be frozen and only a small classifier trained, as in the emotion2vec and wav2vec 2.0 studies. The real costs are collecting consented, representative data, multiple human ratings per clip, evaluation on unseen speakers and a legal review of where the system may be used.
04工作範例
建立並誠實測試小型 SER 分類器
下載 RAVDESS 語音文件,注意 CC BY-NC-SA 4.0 許可證,並從每個檔案名稱讀取情緒標籤。
按演員劃分,而不是按文件劃分,例如演員 1 到 20 用於訓練,21 到 24 用於測試,因此訓練中不會出現測試語音。
提取功能有兩種方式:使用 opensmile Python 套件的 eGeMAPS 描述符,以及來自預訓練檢查點的平均 wav2vec 2.0 層輸出。
在每個特徵集上訓練相同的邏輯迴歸分類器。
報告未加權的平均回憶和每種情緒的混淆矩陣,並記下哪些情緒被混淆,例如平靜和中性。
在一些自然語音片段(例如朋友同意的錄音)上測試更好的模型,並將其標籤與聽眾報告的內容進行比較。
它顯示了什麼
兩個基線模型和一個混淆矩陣表明,學習的語音特徵在表演數據上優於手動設計的特徵。最後一步通常會顯示自然語音的急劇下降,這是信任任何 SER 結果之前最重要的檢查。
現實世界的實施
研究人員在 IEMOCAP 上訓練和測試 SER 系統,IEMOCAP 是在南加州大學錄製的約 12 小時的表演和即興對話,並標有類別、效價、激活和主導性。
MSP-Podcast 語料庫提供超過 400 小時的自然播客語音,每個樣本都由至少 5 個人評分,用於建立在實驗室外工作的模型。
開源的情緒2vec模型於2023年12月發布,提供了預先訓練的情緒表示,只需要頂部的線性分類器,並在10種語言的資料集上顯示了增益。
Interspeech 2025 自然主義語音情緒挑戰賽中的團隊結合了音訊和文字基礎模型;一篇已發表的文章在官方測試集上達到了 39.79% 的宏觀 F1 分數。
根據2025年2月2日起適用的歐盟人工智慧法案第5(1)(f)條,歐盟的雇主或學校不得使用人工智慧來推斷人們的情緒,除非出於醫療或安全原因。
語音情緒辨識的未來
Research is moving from acted to natural speech and from audio-only to multimodal models. The 2025 MSP-Podcast release, with over 400 hours of naturally occurring speech, and the Interspeech 2025 naturalistic challenge built on it show the shift, and the challenge organizers reported that combining audio and text foundation models produced the most robust systems. Emotion-specific self-supervised models such as emotion2vec, which reported gains across 10 languages, aim to reduce the need for large labeled sets. Open problems remain substantial: low accuracy on spontaneous speech, as the roughly 40% macro F1 reported by a 2025 challenge entry illustrates; disagreement among human raters; cultural and language variation; and the question of what a predicted label actually measures. Regulation is a second driver. The EU AI Act's ban on emotion inference in workplaces and schools has applied since February 2, 2025, and products in other settings face scrutiny over consent and fairness. Whether SER becomes reliable enough for high-stakes uses such as health screening, or remains a research and aggregate-analytics tool, is not yet settled.
付諸實踐
為什麼這很重要
- 交通與覆蓋範圍. 它透過轉錄、旁白和語音介面提高了可訪問性。
- 成本與預算. 媒體團隊可以用更少的預算更快地交付精美的音訊。
- 速度與規模. 面向客戶的系統可以處理更大規模的語音互動。
留意
- 如果未徵得同意,語音濫用和冒充風險就會增加。
- 由於口音、方言或嘈雜的環境,準確性可能會下降。
- 如果沒有明確的標籤,合成音訊可能會被誤認為是真實的語音。
後續步驟
- 獲得語音捕獲、克隆和重用的明確同意。
- 測試不同揚聲器和背景條件下的品質。
- 定義人員必須審查或批准輸出的時間。
- 標記合成音訊並保留來源記錄以供問責。
資料來源與延伸閱讀
- University of Southern California, SAIL互動式情感二元運動捕捉 (IEMOCAP) 資料庫
- PLOS ONE瑞爾森情感言語與歌曲視聽資料庫 (RAVDESS)
- arXivMSP 播客語料庫
- arXiv (Interspeech 2021)使用 Wav2vec 2.0 嵌入進行語音情緒識別
- arXiv情緒2vec:語音情緒表示的自我監督預訓練
- ISCA Archive (Interspeech 2025)Interspeech 2025 自然條件下語音情緒辨識挑戰賽
- arXiv利用基於圖的多模態融合和韻律特徵增強語音情緒識別,應對 Interspeech 2025 自然條件下的語音情緒辨識挑戰
- EU Artificial Intelligence Act (Future of Life Institute)第五條:禁止的人工智慧實踐
不斷探索
常見問題
什麼是語音情感辨識?
語音情緒辨識 (SER) 是利用機器學習從說話者聲音的聲學效果來估計說話者的情緒,例如憤怒、悲傷、快樂或中性,或者他們聽起來有多興奮和積極。它對於呼叫分析、健康研究和語音助理很重要,但自然語音的準確性較低,歐盟自 2025 年 2 月起禁止在工人和學生中使用它。
語音情緒辨識主要分析什麼?
SER 關注的是說話的方式——韻律和聲學特徵,如音高、能量和節奏——而不是單字本身。
語音情緒模型可以使用哪些聲學特性作為線索?
SER 研究使用韻律和聲學特徵,但這些線索並不能唯一地映射到所有說話者和上下文中的情緒。
為什麼標記情感數據特別困難?
由於情緒感知是主觀的且隨文化變化,註釋者經常意見不一致,從而創建限制模型準確性的嘈雜標籤。
下列哪一個是著名的情緒語音資料集?
IEMOCAP(以及 RAVDESS 和 CREMA-D)是語音情緒辨識的標準基準;其他是圖像或文字資料集。
現代 SER 系統通常如何利用有限的標記資料?
在大型未標記語音(例如 wav2vec 2.0、HuBERT)上預訓練的自監督模型可以很好地轉移到具有小型標記資料集的情緒任務。