声音事件检测

Sound event detection asks two questions about an audio recording: what sounds are present, and when does each one occur. The output is a list of events, each with a class label, an onset (start time) and an offset (end time). The sed_eval documentation, a widely used evaluation toolbox, describes the task as locating and classifying sounds, estimating onset and offset for each sound event instance and giving it a textual label. Real recordings are usually polyphonic, meaning several events overlap: a person talks while a dishwasher runs and a dog barks. A detector must report all of them at once.

SED sits between two related tasks. Audio tagging, also called audio classification, only says which classes appear somewhere in a clip, with no timing. Acoustic scene classification gives a single label to a whole recording, such as 'street' or 'office'. SED adds temporal localization to tagging, which is what applications like alarms, captioning and audio search need.

Most modern systems follow the same pipeline. First, audio is converted into a log-Mel spectrogram, a time-frequency picture scaled to roughly match human pitch perception. Second, a neural network, commonly a convolutional recurrent neural network (CRNN), reads the spectrogram. The convolutional layers detect local patterns such as the harmonics of a bark, and the recurrent layers track context over time. Third, the network outputs a probability for every class at every time frame. Fourth, post-processing turns those frame probabilities into events: apply a threshold, smooth with a median filter to remove flicker, and merge consecutive active frames into one event with an onset and offset.

The main practical obstacle is labeling. Marking exact start and end times by hand is slow, so most available data has weak labels, meaning clip-level tags without timestamps. AudioSet, released by Google with an ICASSP 2017 paper, contains about 2.08 million human-labeled 10-second YouTube clips across 527 classes, with labels that only verify a sound is present. In 2020 Google added temporally strong labels for 16,996 evaluation clips and 103,463 training clips, about 5% of the training set, using a vocabulary of around 600 labels. Because strong labels are scarce, research has focused on learning timing from weak labels, synthetic data and unlabeled audio.

The DCASE community (Detection and Classification of Acoustic Scenes and Events) has run challenges on this since 2013, and its Task 4 is the main public benchmark for domestic SED. The 2023 edition used DESED, a dataset of 10-second clips from AudioSet or synthesized with the Scaper library, covering 10 household classes. Its development set offered 1,578 weakly labeled clips, 14,412 unlabeled in-domain clips and 10,000 synthetic clips with exact timestamps. The baseline was a mean-teacher CRNN, a semi-supervised method in which a student network learns to match the predictions of a slowly updated teacher on unlabeled audio. A second baseline added frame-level embeddings from BEATs, a model pretrained on AudioSet, and raised the development-set PSDS scenario 1 score from 0.359 to 0.500.

Evaluation choices change results a great deal. Segment-based metrics divide time into fixed segments, often one second, and check whether each class is correctly marked active in each segment. Event-based metrics match whole events, counting a detection as correct if its label matches and its boundaries fall within a tolerance called a collar. DCASE's contrastive event-based F1 uses a 200 millisecond collar on onsets and the larger of 200 milliseconds or 20% of the event length on offsets. The error rate counts substitutions, deletions and insertions relative to the number of reference events, as in speech recognition. These metrics all depend on a chosen decision threshold.

The polyphonic sound detection score (PSDS), proposed by Cagdas Bilen and colleagues in a paper posted October 18, 2019 and presented at ICASSP 2020, addresses that dependence. It sweeps the decision threshold, builds a receiver operating characteristic curve of true positive rate against false positives per hour, and summarizes it as one number. It matches events by how much they intersect rather than by fixed collars, which the authors argue is more robust to subjective labeling. DCASE uses two settings: scenario 1 emphasizes precise timing for fast reactions, and scenario 2 emphasizes avoiding confusion between classes. The 2023 task switched to a fully threshold-independent PSDS implementation.

Limitations remain significant. Reference annotations are subjective: the DCASE organizers note that experts checking AudioSet clips found many did not contain the labeled event. Systems trained on synthetic soundscapes can struggle on real recordings. Short, quiet or heavily overlapped events are often missed, and rare classes have little training data. Reported scores depend on the metric, collar and threshold chosen, so numbers from different papers are often not comparable; DCASE itself notes that its 2023 threshold-independent PSDS is not directly comparable to earlier years. On cost, the 2023 challenge made reporting training energy mandatory and added a count of multiply-accumulate operations per 10 seconds of audio, reflecting concern about compute as models grow.