音訊人工智慧指南

聲音事件偵測

聲音事件檢測(SED)是找出錄音中哪些聲音發生以及每個聲音何時開始和結束的任務,例如狗叫聲從 2.6 秒到 7.2 秒,同時語音與之重疊。

聆聽此指南AI 語音敘述 · ~8 min

本頁閱讀時間 9 分鐘
  1. 概述
  2. 重點摘要
  3. 深入探討
  4. 手動評分檢測器輸出
  5. 現實世界的實施
  6. 聲音事件偵測的未來
  7. 付諸實踐
  8. 資料來源與延伸閱讀
  9. 不斷探索
  10. 常見問題

概述

它為警報、智能家居觸發和音頻搜索提供動力,且比簡單音頻標記更難,因為系統必須為重疊聲音設置時間邊界,往往在學習缺乏時間戳的標籤時完成。

重點摘要

  1. 聲音事件檢測輸出帶有開始和結束時間的標記事件,而音頻標記僅報告片段中出現的聲音。
  2. AudioSet 擁有約 208 萬個弱標記的 10 秒剪輯,涵蓋 527 個類別;強時間標記僅存在於少量子集。
  3. DCASE 的對比事件 F1 使用 200 毫秒的起始邊界和 200 毫秒或 20% 事件長度的終止邊界。
  4. PSDS,於 2019 年提出,掃描決策閾值並通過交集匹配事件,因此不依賴單一操作點。
  5. 將預訓練的 BEATs 嵌入加入 DCASE 2023 基準模型,使開發集 PSDS 場景 1 的得分從 0.359 提升至 0.500。

深入探討

Sound event detection asks two questions about an audio recording: what sounds are present, and when does each one occur. The output is a list of events, each with a class label, an onset (start time) and an offset (end time). The sed_eval documentation, a widely used evaluation toolbox, describes the task as locating and classifying sounds, estimating onset and offset for each sound event instance and giving it a textual label. Real recordings are usually polyphonic, meaning several events overlap: a person talks while a dishwasher runs and a dog barks. A detector must report all of them at once.

SED sits between two related tasks. Audio tagging, also called audio classification, only says which classes appear somewhere in a clip, with no timing. Acoustic scene classification gives a single label to a whole recording, such as 'street' or 'office'. SED adds temporal localization to tagging, which is what applications like alarms, captioning and audio search need.

Most modern systems follow the same pipeline. First, audio is converted into a log-Mel spectrogram, a time-frequency picture scaled to roughly match human pitch perception. Second, a neural network, commonly a convolutional recurrent neural network (CRNN), reads the spectrogram. The convolutional layers detect local patterns such as the harmonics of a bark, and the recurrent layers track context over time. Third, the network outputs a probability for every class at every time frame. Fourth, post-processing turns those frame probabilities into events: apply a threshold, smooth with a median filter to remove flicker, and merge consecutive active frames into one event with an onset and offset.

The main practical obstacle is labeling. Marking exact start and end times by hand is slow, so most available data has weak labels, meaning clip-level tags without timestamps. AudioSet, released by Google with an ICASSP 2017 paper, contains about 2.08 million human-labeled 10-second YouTube clips across 527 classes, with labels that only verify a sound is present. In 2020 Google added temporally strong labels for 16,996 evaluation clips and 103,463 training clips, about 5% of the training set, using a vocabulary of around 600 labels. Because strong labels are scarce, research has focused on learning timing from weak labels, synthetic data and unlabeled audio.

The DCASE community (Detection and Classification of Acoustic Scenes and Events) has run challenges on this since 2013, and its Task 4 is the main public benchmark for domestic SED. The 2023 edition used DESED, a dataset of 10-second clips from AudioSet or synthesized with the Scaper library, covering 10 household classes. Its development set offered 1,578 weakly labeled clips, 14,412 unlabeled in-domain clips and 10,000 synthetic clips with exact timestamps. The baseline was a mean-teacher CRNN, a semi-supervised method in which a student network learns to match the predictions of a slowly updated teacher on unlabeled audio. A second baseline added frame-level embeddings from BEATs, a model pretrained on AudioSet, and raised the development-set PSDS scenario 1 score from 0.359 to 0.500.

Evaluation choices change results a great deal. Segment-based metrics divide time into fixed segments, often one second, and check whether each class is correctly marked active in each segment. Event-based metrics match whole events, counting a detection as correct if its label matches and its boundaries fall within a tolerance called a collar. DCASE's contrastive event-based F1 uses a 200 millisecond collar on onsets and the larger of 200 milliseconds or 20% of the event length on offsets. The error rate counts substitutions, deletions and insertions relative to the number of reference events, as in speech recognition. These metrics all depend on a chosen decision threshold.

The polyphonic sound detection score (PSDS), proposed by Cagdas Bilen and colleagues in a paper posted October 18, 2019 and presented at ICASSP 2020, addresses that dependence. It sweeps the decision threshold, builds a receiver operating characteristic curve of true positive rate against false positives per hour, and summarizes it as one number. It matches events by how much they intersect rather than by fixed collars, which the authors argue is more robust to subjective labeling. DCASE uses two settings: scenario 1 emphasizes precise timing for fast reactions, and scenario 2 emphasizes avoiding confusion between classes. The 2023 task switched to a fully threshold-independent PSDS implementation.

Limitations remain significant. Reference annotations are subjective: the DCASE organizers note that experts checking AudioSet clips found many did not contain the labeled event. Systems trained on synthetic soundscapes can struggle on real recordings. Short, quiet or heavily overlapped events are often missed, and rare classes have little training data. Reported scores depend on the metric, collar and threshold chosen, so numbers from different papers are often not comparable; DCASE itself notes that its 2023 threshold-independent PSDS is not directly comparable to earlier years. On cost, the 2023 challenge made reporting training energy mandatory and added a count of multiply-accumulate operations per 10 seconds of audio, reflecting concern about compute as models grow.

04工作範例

手動評分檢測器輸出

  1. 為 10 秒剪輯編寫參考:0.0 到 4.0 秒的語音和 2.6 到 3.4 秒的狗吠聲。

  2. 編寫系統輸出:0.1 到 4.3 秒的語音,2.7 到 3.5 秒的狗,以及 6.0 到 6.5 秒的錯誤「攪拌機」事件。

  3. 應用基於事件的匹配和 200 毫秒起始項圈:語音起始偏移 0.1 秒,狗偏移 0.1 秒,因此兩個起始都匹配。

  4. 使用 200 毫秒或事件長度 20% 中較大者的項圈檢查偏移:語音允許 0.8 秒,偏差 0.3; dog 允許 0.2,偏差 0.1,所以兩者匹配。

  5. 計數結果:兩個真陽性、一個假陽性(混合器)和零個假陰性。

  6. 計算精準度 2/3,約 0.67,召回率 2/2 = 1.0,F 分數 2 x 0.67 x 1.0 / 1.67,約 0.80。

它顯示了什麼

該剪輯基於事件的 F 分數約為 0.80。透過將相同的參考和輸出清單輸入到 sed_eval EventBasedMetrics 類別(具有 0.2 秒領和 20% 偏移選項)來檢查手動計算,這應該報告相同的計數。

現實世界的實施

必須快速反應的家庭監控,例如在出現聲音時觸發警報或調整家庭自動化,這是 DCASE 2023 任務 4「場景 1」評估背後的用例。

在 2020 年以來 DCASE 挑戰賽中使用的 10 秒家庭剪輯的 DESED 資料集中偵測 10 種家庭聲音類別,包括狗叫聲、警鈴和跑菜聲。

使用 Google 的 AudioSet 強標籤,按聲音類型和時間搜尋大型音訊檔案,該標籤標記了超過 120,000 個 YouTube 剪輯中事件的開始和結束時間。

使用開源 sed_eval 工具箱評估偵測器,該工具箱使用基於段和基於事件的精度、召回率、F 分數和錯誤率,根據參考註釋對系統輸出進行評分。

重複使用預先訓練的音訊網路(例如 PANN)作為時間分辨偵測器的特徵擷取器,其在 AudioSet 標記上的平均精度達到 0.439。

聲音事件偵測的未來

Several research directions are visible in the DCASE task design and the datasets around it. The 2023 Task 4 asked how pretrained embeddings, external data and model complexity affect results, and its BEATs-based baseline showed large gains from models pretrained on AudioSet, which points toward general audio foundation models adapted to detection. Labeling remains the bottleneck: Google's 2021 strong-label release covered only about 5% of AudioSet's training clips, so methods that learn timing from weak, noisy or partial labels are still central. Evaluation itself is an open problem. PSDS (2019 to 2020) and later threshold-independent implementations (2022) were introduced because collar-based F-scores depend on arbitrary thresholds, but each change makes scores incomparable with older work. The DCASE 2023 decision to require energy reporting and operation counts suggests efficiency will be weighed alongside accuracy, which matters for always-on devices. Open questions include detecting rare or unseen classes, handling heavily overlapped sounds, and transferring systems trained on synthetic or YouTube audio to real homes, cities and wildlife recordings.

付諸實踐

為什麼這很重要

  • 交通與覆蓋範圍. 它透過轉錄、旁白和語音介面提高了可訪問性。
  • 成本與預算. 媒體團隊可以用更少的預算更快地交付精美的音訊。
  • 速度與規模. 面向客戶的系統可以處理更大規模的語音互動。

留意

  • 如果未徵得同意,語音濫用和冒充風險就會增加。
  • 由於口音、方言或嘈雜的環境,準確性可能會下降。
  • 如果沒有明確的標籤,合成音訊可能會被誤認為是真實的語音。

後續步驟

  1. 獲得語音捕獲、克隆和重用的明確同意。
  2. 測試不同揚聲器和背景條件下的品質。
  3. 定義人員必須審查或批准輸出的時間。
  4. 標記合成音訊並保留來源記錄以供問責。

資料來源與延伸閱讀

  1. DCASE Community使用弱標籤和合成音景的聲音事件偵測(DCASE 2023 任務 4A)
  2. arXiv (ICASSP 2020)聲音事件偵測的穩健評估框架
  3. Tampere University Audio Research Groupsed_eval:聲音事件偵測指標
  4. Google ResearchAudioSet
  5. Google ResearchAudioSet 臨時強標籤下載(2021 年 5 月)
  6. arXivPANNs:用於音訊模式識別的大規模預訓練音訊神經網路

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

參加聲音事件偵測測驗

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

什麼是聲音事件偵測?

聲音事件偵測 (SED) 的任務是尋找錄音中出現的聲音以及每個聲音的確切開始和停止時間,例如狗在 2.6 秒到 7.2 秒內吠叫,而語音與之重疊。它為警報、智慧家庭觸發器和音訊搜尋提供動力,並且比簡單的音訊標記更難,因為系統必須在重疊的聲音上設置時間界限,通常同時從缺乏時間戳的標籤中學習。

除了片段層級的標籤外,時間聲音事件偵測還估計了哪些額外的輸出?

該任務增加了估計的時間定位,其準確性是經過評估的,而非保證的。

為什麼聲音事件檢測經常被描述為複調問題?

現實世界的音訊經常同時包含多個重疊事件,因此模型必須預測每個畫面的多個活動標籤。

SED 訓練資料中的「弱標籤」是什麼意思?

弱標籤表明剪輯中存在沒有確切開始和偏移時間的事件,這使得精確的時間學習變得更加困難。

在引用的 DCASE 2024 基準中指出了哪種分類器結構?

該基準模型在其既定設定下結合了時頻特徵處理與時間建模。

此處描述的記錄聲音偵測基準情境中使用了哪種時頻表示法?

這是一種指定的特徵選擇,而非對每個可能的偵測器的要求。