Diarization Agbọrọsọ

Speaker diarization is the task of working out who spoke when in an audio recording. The output is a list of time segments, each tagged with a speaker label such as SPEAKER_00 or SPEAKER_01. The labels are anonymous: diarization says that the same voice returns at 3:15 and 7:40, not that the voice belongs to a particular person. Naming speakers is a separate task, speaker identification, which compares voices with enrolled samples. Diarization is usually paired with automatic speech recognition, so a transcript of a meeting or phone call can show turns of dialogue instead of one undivided block of text.

The standard approach is a pipeline. Hervé Bredin's Interspeech 2023 paper on pyannote.audio 2.1 describes three stages that are typical of modern systems. First, a segmentation model slides a short window over the audio and decides, frame by frame, whether anyone is speaking and how many local speakers are active. This folds in voice activity detection, separating speech from silence, music and noise. Second, a speaker embedding model turns each local speaker's speech into a fixed-length vector, a list of numbers designed so that two clips of the same voice land close together and clips of different voices land far apart. Third, a clustering algorithm groups those vectors across the whole recording, so speaker 2 in minute one and speaker 1 in minute ten can be recognized as the same person. Agglomerative clustering, which starts with every segment as its own group and merges the closest pairs, is the method pyannote uses.

Overlapping speech, when two people talk at once, is the hardest part. Older pipelines assigned each moment to one speaker and simply lost the second voice. In 2019 Yusuke Fujita, Shinji Watanabe and colleagues proposed end-to-end neural diarization (EEND), a single network that outputs, for every frame, an independent yes or no for each speaker, which allows overlap. Because speaker labels have no natural order, they trained it with a permutation-free objective that scores the best matching between predicted and true speakers. On simulated mixtures it reached 12.28% diarization error rate against 28.77% for a clustering-based baseline.

pyannote's current segmentation model takes a further step. Alexis Plaquet and Hervé Bredin's 2023 paper replaced the per-speaker yes or no outputs with powerset classification, in which each class is a specific combination of active speakers. The segmentation-3.0 model card lists seven classes for each 10-second chunk: no speech, each of three speakers alone, and each of the three possible pairs. The authors report better results on 9 benchmarks, mostly on overlapping speech, and removal of the detection threshold that the older formulation depended on. The speaker-diarization-3.1 pipeline combines this model with embeddings and clustering, runs in pure PyTorch, and is released under the MIT license.

Performance is measured with diarization error rate (DER), the share of reference speaking time that is wrongly handled. According to the dscore toolkit, which follows NIST's RT-09 evaluation plan, DER is the sum of three errors: missed speech, false alarm speech where silence was labeled as speech, and speaker confusion where speech went to the wrong speaker. Before scoring, system labels are mapped to reference labels so the arbitrary names do not count against the system. Two settings change results a lot: a forgiveness collar, a short window around each boundary that is not scored, and whether overlapping speech is included. pyannote reports its numbers with no collar and with overlap scored, the strictest setup.

Those strict numbers show how much the domain matters. The speaker-diarization-3.1 model card reports DER of 7.8% on the REPERE broadcast set, 11.3% on VoxConverse, 12.2% on the AISHELL-4 meeting corpus, 18.8% on AMI headset mixes, 21.7% on DIHARD 3 and 50.0% on AVA-AVD, a set of movie clips. A September 2025 independent benchmark by Lanzendörfer and colleagues evaluated five systems on 196.6 hours of audio in five languages and found the commercial PyannoteAI model best at 11.2% DER, with the open-source DiariZen at 13.3%. It found most errors came from missed speech, followed by speaker confusion, and that recordings with many speakers were hardest.

Common failure modes follow from the pipeline. Short interjections such as yeah or right are often missed or given to the wrong person, because a fraction of a second carries little voice information. Similar voices, such as relatives, get merged. A single speaker who changes microphone, moves across a room or raises their voice can be split into two. Background music, television and crosstalk create false alarms. Errors also compound: the 2025 benchmark notes that diarization mistakes spread to downstream uses such as transcription and summaries, so a confused speaker turn can put words in the wrong mouth.

In practice, pyannote pipelines run in PyTorch and can be moved to a GPU with one call, which matters for long files. pyannote's pipelines are gated on Hugging Face, so users must accept the conditions and use an access token before downloading. Audio is downmixed to mono and resampled to 16 kHz on loading. Supplying the number of speakers when it is known, or a minimum and maximum, can reduce clustering mistakes. Because accuracy varies so much by domain, the reliable way to choose a system is to hand-label a few minutes of your own audio and measure DER with a fixed collar and overlap setting.