Audio AI-GIDS

Spreker Diarisatie

Speaker diarization splits a recording into time segments and labels each one by speaker, answering the question of who spoke when without needing to know anyone's name.

Luister naar deze gidsAI-verteld · ~7 min

Op deze pagina8 min leestijd
  1. Overzicht
  2. Belangrijkste inzichten
  3. Diepe duik
  4. Diarizing and scoring a five-minute meeting clip
  5. Implementatie in de echte wereld
  6. De toekomst van sprekersdiarisatie
  7. Het in de praktijk brengen
  8. Bronnen en verdere literatuur
  9. Blijf verkennen
  10. Veelgestelde vragen

Overzicht

It turns raw transcripts of meetings, calls and interviews into readable dialogue, and it is measured by diarization error rate, which still ranges from about 8% to 50% across public benchmarks for a leading open-source pipeline.

Belangrijkste inzichten

  1. Speaker diarization labels who spoke when in a recording with anonymous speaker tags; naming the speakers is a separate identification task.
  2. Typical pipelines run local segmentation, speaker embeddings and clustering, as described for pyannote.audio in a 2023 Interspeech paper.
  3. Diarization error rate adds missed speech, false alarms and speaker confusion, divided by total reference speech time.
  4. pyannote's speaker-diarization-3.1 reports DER from 7.8% on broadcast news to 50.0% on movie clips under strict scoring with no collar.
  5. A 2025 independent benchmark found missed speech was the largest error source and recordings with many speakers were the hardest.

Diepe duik

Speaker diarization is the task of working out who spoke when in an audio recording. The output is a list of time segments, each tagged with a speaker label such as SPEAKER_00 or SPEAKER_01. The labels are anonymous: diarization says that the same voice returns at 3:15 and 7:40, not that the voice belongs to a particular person. Naming speakers is a separate task, speaker identification, which compares voices with enrolled samples. Diarization is usually paired with automatic speech recognition, so a transcript of a meeting or phone call can show turns of dialogue instead of one undivided block of text.

The standard approach is a pipeline. Hervé Bredin's Interspeech 2023 paper on pyannote.audio 2.1 describes three stages that are typical of modern systems. First, a segmentation model slides a short window over the audio and decides, frame by frame, whether anyone is speaking and how many local speakers are active. This folds in voice activity detection, separating speech from silence, music and noise. Second, a speaker embedding model turns each local speaker's speech into a fixed-length vector, a list of numbers designed so that two clips of the same voice land close together and clips of different voices land far apart. Third, a clustering algorithm groups those vectors across the whole recording, so speaker 2 in minute one and speaker 1 in minute ten can be recognized as the same person. Agglomerative clustering, which starts with every segment as its own group and merges the closest pairs, is the method pyannote uses.

Overlapping speech, when two people talk at once, is the hardest part. Older pipelines assigned each moment to one speaker and simply lost the second voice. In 2019 Yusuke Fujita, Shinji Watanabe and colleagues proposed end-to-end neural diarization (EEND), a single network that outputs, for every frame, an independent yes or no for each speaker, which allows overlap. Because speaker labels have no natural order, they trained it with a permutation-free objective that scores the best matching between predicted and true speakers. On simulated mixtures it reached 12.28% diarization error rate against 28.77% for a clustering-based baseline.

pyannote's current segmentation model takes a further step. Alexis Plaquet and Hervé Bredin's 2023 paper replaced the per-speaker yes or no outputs with powerset classification, in which each class is a specific combination of active speakers. The segmentation-3.0 model card lists seven classes for each 10-second chunk: no speech, each of three speakers alone, and each of the three possible pairs. The authors report better results on 9 benchmarks, mostly on overlapping speech, and removal of the detection threshold that the older formulation depended on. The speaker-diarization-3.1 pipeline combines this model with embeddings and clustering, runs in pure PyTorch, and is released under the MIT license.

Performance is measured with diarization error rate (DER), the share of reference speaking time that is wrongly handled. According to the dscore toolkit, which follows NIST's RT-09 evaluation plan, DER is the sum of three errors: missed speech, false alarm speech where silence was labeled as speech, and speaker confusion where speech went to the wrong speaker. Before scoring, system labels are mapped to reference labels so the arbitrary names do not count against the system. Two settings change results a lot: a forgiveness collar, a short window around each boundary that is not scored, and whether overlapping speech is included. pyannote reports its numbers with no collar and with overlap scored, the strictest setup.

Those strict numbers show how much the domain matters. The speaker-diarization-3.1 model card reports DER of 7.8% on the REPERE broadcast set, 11.3% on VoxConverse, 12.2% on the AISHELL-4 meeting corpus, 18.8% on AMI headset mixes, 21.7% on DIHARD 3 and 50.0% on AVA-AVD, a set of movie clips. A September 2025 independent benchmark by Lanzendörfer and colleagues evaluated five systems on 196.6 hours of audio in five languages and found the commercial PyannoteAI model best at 11.2% DER, with the open-source DiariZen at 13.3%. It found most errors came from missed speech, followed by speaker confusion, and that recordings with many speakers were hardest.

Common failure modes follow from the pipeline. Short interjections such as yeah or right are often missed or given to the wrong person, because a fraction of a second carries little voice information. Similar voices, such as relatives, get merged. A single speaker who changes microphone, moves across a room or raises their voice can be split into two. Background music, television and crosstalk create false alarms. Errors also compound: the 2025 benchmark notes that diarization mistakes spread to downstream uses such as transcription and summaries, so a confused speaker turn can put words in the wrong mouth.

In practice, pyannote pipelines run in PyTorch and can be moved to a GPU with one call, which matters for long files. pyannote's pipelines are gated on Hugging Face, so users must accept the conditions and use an access token before downloading. Audio is downmixed to mono and resampled to 16 kHz on loading. Supplying the number of speakers when it is known, or a minimum and maximum, can reduce clustering mistakes. Because accuracy varies so much by domain, the reliable way to choose a system is to hand-label a few minutes of your own audio and measure DER with a fixed collar and overlap setting.

04Uitgewerkt voorbeeld

Diarizing and scoring a five-minute meeting clip

  1. Pick a five-minute recording with three or four speakers and some crosstalk, and convert it to 16 kHz mono WAV.

  2. Listen through once and hand-label speaker turns in an RTTM file, the plain-text format diarization scorers read, including overlapping regions.

  3. Accept the model conditions on Hugging Face, then run pyannote's speaker-diarization-3.1 pipeline on the clip and save its output as RTTM.

  4. Score the output against your labels with dscore, once with no collar and once with a 0.25 second collar, and note the missed speech, false alarm and confusion parts.

  5. Re-run the pipeline with the true number of speakers supplied and score it again.

  6. Listen to the segments with the largest errors and classify them as overlap, short interjection or speaker mix-up.

Wat het laat zien

Two or three DER figures and a list of error types for your own audio. Expect the collar to lower DER and a known speaker count to reduce confusion; if most error is missed speech in overlaps, that matches the 2025 benchmark's finding.

Implementatie in de echte wereld

Producing a speaker-labeled meeting transcript by running pyannote's open-source speaker-diarization-3.1 pipeline, which accepts 16 kHz mono audio and needs no manual speaker count, alongside a speech recognizer.

Adding Speaker 1 and Speaker 2 labels to Whisper output with WhisperX, which calls a pyannote diarization model and attaches its speaker labels to the word-timed transcript.

Scoring a diarization system against hand labels with the dscore toolkit, which computes diarization error rate using NIST's md-eval scoring and the Jaccard error rate introduced for the DIHARD II challenge.

Building speaker-attributed speech-to-text with NVIDIA's Sortformer, a 2024 model released in the NeMo framework that feeds speaker labels into a transcription model's encoder.

Comparing systems across meeting, broadcast and in-the-wild audio using public sets such as AMI, AISHELL-4, DIHARD 3 and VoxConverse, the benchmarks reported on pyannote's model card.

De toekomst van sprekersdiarisatie

Several research directions are visible in recent primary sources. The September 2025 benchmark by Lanzendörfer and colleagues concluded that diarization remains unsolved, with missed speech the largest error source and many-speaker recordings the hardest, which keeps attention on overlap and detection rather than only clustering. Joint modeling of transcription and speakers is a second direction: NVIDIA's Sortformer, published at ICML 2025, feeds speaker labels into a speech-to-text encoder so that words and speaker turns are predicted together rather than stitched afterward. A third is evaluation itself. The gap between pyannote's 7.8% DER on broadcast news and 50.0% on movie clips shows that single benchmark numbers say little about a new domain, and scoring choices such as collars change results. Multilingual and far-field recordings, many-speaker scenes and long recordings with speakers who change acoustically over time remain difficult. Whether end-to-end models will fully replace modular pipelines is an open question, since current production tools still combine segmentation, embeddings and clustering.

Het in de praktijk brengen

Waarom het ertoe doet

  • Toegang en bereik. Het verbetert de toegankelijkheid via transcriptie, gesproken tekst en spraakinterfaces.
  • Kosten en budget. Mediateams kunnen met kleinere budgetten sneller gepolijste audio leveren.
  • Snelheid en schaal. Klantgerichte systemen kunnen gesproken interacties op grotere schaal verwerken.

Pas op voor

  • Het risico op stemmisbruik en imitatie neemt toe als de toestemming ontbreekt.
  • De nauwkeurigheid kan afnemen bij accenten, dialecten of luidruchtige omgevingen.
  • Synthetische audio kan worden aangezien voor authentieke spraak zonder duidelijke labels.

Volgende stappen

  1. Verkrijg expliciete toestemming voor het vastleggen, klonen en hergebruiken van spraak.
  2. Test de kwaliteit van diverse sprekers en achtergrondomstandigheden.
  3. Bepaal wanneer een mens de output moet beoordelen of goedkeuren.
  4. Label synthetische audio en houd de herkomstgegevens bij voor verantwoording.

Bronnen en verdere literatuur

  1. Hugging Face (pyannote)pyannote/speaker-diarization-3.1
  2. Hugging Face (pyannote)pyannote/segmentation-3.0
  3. ISCA (Interspeech 2023)pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe
  4. arXiv (Interspeech 2023)Powerset multi-class cross entropy loss for neural speaker diarization
  5. arXiv (Interspeech 2019)End-to-End Neural Speaker Diarization with Permutation-Free Objectives
  6. arXiv (ICML 2025)Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text Systems
  7. GitHubnryant/dscore: Diarization scoring tools
  8. arXivBenchmarking Diarization Models

Blijf verkennen

Gratis nieuwsbrief

Ontvang de dagelijkse AI-briefing

Drie verifieerde AI-verhalen elke weekdagmorgen, geschreven in eenvoudig Engels. Gratis voor altijd, geen advertenties.

Eén e-mail elke weekdag. Afmelden met één klik. We verkopen of delen je adres nooit.

Test jezelf

Take the Speaker Diarization quiz

Directe feedback op elke antwoord, en een deelbare certificaat met een verifieerbare ID zodra je een cursus hebt afgerond.

Quiz starten

Ondersteun gratis AI-onderwijs. AI Understanding is een 501(c)(3) non-profit — geen advertenties, geen betaalmuur, nooit. Doe een donatie

Veelgestelde vragen

What is Speaker Diarization?

Speaker diarization splits a recording into time segments and labels each one by speaker, answering the question of who spoke when without needing to know anyone's name. It turns raw transcripts of meetings, calls and interviews into readable dialogue, and it is measured by diarization error rate, which still ranges from about 8% to 50% across public benchmarks for a leading open-source pipeline.

Met welke vraag wordt het dagboekschrijven van sprekers vaak samengevat?

Diarisatie verdeelt de audio in de loop van de tijd per spreker, waarbij wordt geantwoord "wie wanneer heeft gesproken" in plaats van de woorden zelf te transcriberen.

Wat is een luidsprekerinbedding?

Een luidsprekerinbedding is een vector met een vaste lengte waarbij fragmenten van dezelfde persoon dicht bij elkaar staan, waardoor clustering op basis van identiteit mogelijk wordt.

Welke maatstaf wordt vaak gebruikt om dagboekregistratiesystemen te evalueren?

DER combineert gemiste spraak, valse alarmen en verwarring van de spreker in één percentage, waardoor dit de standaardmaatstaf voor dagboekregistratie wordt.

Moet bij standaard dagboekschrijven vooraf de echte naam van de sprekers bekend zijn?

Diarisatie bundelt stemmen en kent anonieme labels toe; het vereist niet dat je weet wie de mensen daadwerkelijk bij naam zijn.

Welk probleem lost permutatie-invariante training op bij end-to-end neurale diarisatie?

Speaker-output indices are arbitrary in diarization. Permutation-invariant training finds a matching between predicted and reference speaker streams so the loss does not depend on their index order; this is distinct from representing overlap with multiple active labels.