Toosinta Qasabka ah

Forced alignment is the job of placing a known transcript onto a recording in time. The inputs are an audio file and the text of what was said. The output is a start and end time for each word, and often for each phone, the smallest unit of sound that tells words apart, such as the k sound at the start of cat. It is called forced because the system is not allowed to decide what was said: it must explain the audio using exactly the given words in the given order. That separates it from speech recognition, which has to guess the words. With the word sequence fixed, the search is much smaller, so an aligner can be precise about timing. It also means an aligner does not check the transcript. If the text is wrong, the wrong words still get placed somewhere.

The classic method uses a hidden Markov model (HMM), a statistical model that treats speech as a walk through a chain of hidden states. The Montreal Forced Aligner (MFA) documentation describes forced alignment as turning an orthographic transcript into a time-aligned version by using a pronunciation dictionary to look up the phones for each word. Step one is that lookup: each word becomes a phone sequence, and words missing from the dictionary can be handled by a grapheme-to-phoneme model that guesses pronunciations from spelling. Step two converts the audio into acoustic features. According to the 2024 Interspeech comparison by Rousso and colleagues, MFA uses 39 mel-frequency cepstral coefficient (MFCC) features, compact summaries of the spectrum's shape, computed every 10 ms over a 25 ms window.

Step three strings the phones of every word into one long left-to-right chain of HMM states, usually with optional silence between words. A Gaussian mixture model (GMM) scores how well each 10 ms frame of features matches each state. Step four runs the Viterbi algorithm, a dynamic programming search that finds the single most probable way to assign every frame to a state while moving through the chain in order. Wherever the best path crosses from one phone to the next is a phone boundary, and word boundaries follow from the phones. The resolution is therefore one frame, 10 ms.

MFA trains its acoustic models in passes, which the user guide lists as monophone models that ignore context, triphone models that consider the phone on each side, an LDA plus MLLT feature transform that makes phones easier to tell apart, and a final pass with per-speaker adaptation. Because training needs only transcripts, not hand-marked boundaries, MFA can be trained on the very data it is asked to align. The 2017 Interspeech paper by Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner and Morgan Sonderegger introduced MFA as an update of the earlier Prosodylab-Aligner, moving from the HTK toolkit to Kaldi, adding triphones and speaker adaptation, and enabling parallel processing. It reported better boundaries than Prosodylab-Aligner and FAVE on English conversational and laboratory speech, and found that training on the target data generally helped. The documentation is now at version 3.4, which adds model formats that work with Hugging Face.

Neural aligners arrived with models trained using connectionist temporal classification (CTC), a training objective that lets a network learn to output characters from audio without being told where each character occurs. Wav2vec 2.0 models produce one output frame every 20 ms, per the Rousso paper. Meta's Massively Multilingual Speech (MMS) paper, posted in May 2023, built a single recognizer for 1,107 languages, and torchaudio packages an MMS model as MMS_FA for alignment. In March 2023 Max Bain, Jaesung Huh, Tengda Han and Andrew Zisserman posted WhisperX, accepted at Interspeech 2023, which adds voice activity detection and forced phoneme alignment to Whisper to get word timestamps, and reports a twelve-fold transcription speedup from batching pre-segmented audio.

The two families can be compared directly. Rousso, Cohen, Keshet and Chodroff scored MFA, WhisperX and MMS on TIMIT, read speech, and Buckeye, conversational speech, counting only words each neural system recognized correctly. Within 25 ms of the hand-labeled boundary, MFA placed 72.8% of TIMIT word boundaries, against 52.7% for WhisperX and 43.5% for MMS. On Buckeye the figures were 69.9%, 43.1% and 52.7%. At a looser 100 ms tolerance the TIMIT gap shrank to 97.4%, 94.2% and 94.7%. The authors suggest CTC training without aligned data and the coarser 20 ms frame rate as likely causes, though they did not test these directly. The practical trade-off is that neural pipelines need no pronunciation dictionary and can produce a transcript when none exists, while HMM aligners give tighter boundaries but need a lexicon and acoustic model for each language.

Failure modes are predictable. Transcripts that leave out filled pauses, false starts or repeated words force the aligner to stretch neighboring words over the extra sound. The WhisperX README says overlapping speech is not handled particularly well, and that tokens containing characters missing from the alignment dictionary, such as 2014. or a price in pounds, get no timing at all, so numbers and symbols should be written out as words first. The torchaudio tutorial warns that CTC models are peaky, meaning a character's probability spike may cover only one or two frames rather than its full duration, and that blank frames make word edges ambiguous. In the 2024 comparison, MFA and WhisperX were both less accurate on conversational Buckeye than on read TIMIT.

Requirements differ by tool. MFA is a command-line package built on Kaldi that needs a pronunciation dictionary and acoustic model for the language, either downloaded or trained on your own corpus. WhisperX targets GPUs; its README claims about 70 times real-time speed with the large-v2 model and under 8 GB of GPU memory with a beam size of 5. The torchaudio alignment APIs were deprecated in version 2.8 and removed in 2.9, so code based on the tutorial needs a pinned version or a replacement. In every case the honest check is to compare a sample of output boundaries with hand-marked ones before trusting the timings.