Less Spiky CTC Alignment for Data Cleaning (2025–)


Motivation

  • Open speech foundation models such as OWSM rely on large web-crawled corpora, whose transcripts are often misaligned with the audio.
  • OWSM v4 cleans the YODAS corpus with CTC alignment [paper]
    • Audio and text are re-aligned with CTC segmentation [paper] using a pre-trained OWSM-CTC model, which splits long recordings into utterances of up to 30 seconds.
    • Each utterance gets a CTC confidence score, and the lowest-scoring 10% within each language are discarded (along with LID-based filtering).
  • So, the quality of the cleaned data depends directly on how accurate the CTC alignment is.

Problem: CTC is Spiky

  • CTC models output sharp, single-frame spikes for each token and predict blank for most of the frames.

    Illustration with synthetic posteriors. Shaded regions are the ground-truth phone segments.
  • This is harmless for recognition, since the blanks are removed anyway, but the spike positions say little about where each token starts and ends.
  • Zeyer et al. show that this peaky behavior comes from the CTC training criterion itself, not from the model architecture [paper]
    • Starting from a uniform initialization, gradient descent on the CTC loss converges to peaky (and sometimes suboptimal) local optima.
    • The blank label is what makes CTC converge well, but it is also what absorbs most of the frames.
    • Training with a label prior removes the peaky behavior and improves convergence.

Previous Work: Label Priors

  • Huang et al. apply label priors to CTC forced alignment [paper]
    • During training, each frame’s posterior is divided by the label prior, ytπt / P(πt)α, so that alignment paths with fewer blanks get higher scores. (α = 0 is the standard CTC; α = 0.3 worked best.)
    • The priors start from a uniform distribution and are re-estimated at the end of every epoch by averaging the model’s posteriors over all frames.
    • The priors are also used in the Viterbi search for the alignment. Applying them only at decoding time, without training, did not help.
  • Results (phoneme-level TDNN model trained on LibriSpeech)
    • The blank prior on training data drops from 0.80 to 0.32, and the average phone duration grows from 21 ms to 64 ms. (Ground truth: 82 ms)
    • Phoneme and word boundary errors are reduced by 12–40% on Buckeye and TIMIT compared to standard CTC and a heuristic offset baseline.
    • It is close to the Montreal Forced Aligner on Buckeye but still behind it on TIMIT, while being much simpler to train and faster to run.

Proposed: BIE-CTC, an HMM-like Topology

  • Label priors are a soft fix: they lower the blank bias, but the blank token itself is still ambiguous.
    • In CTC, blank means both “silence” and “the previous token is still going on.”
    • Also, repeated tokens (e.g., “e, e” in “see”) can only be separated by a blank.
  • Instead of a blank, each token gets an explicit 3-state structure, like the classic 3-state HMM.
    • B-x: the beginning (onset) of token x, with a self-loop of at most 1–2 frames
    • I-x: the inside (steady state) of token x, with an unrestricted self-loop
    • E-x: the end (offset) of token x, with a self-loop of at most 1–2 frames
    • <sil>: an explicit silence token without phases, taking over the “silence” meaning of blank
  • The graph for “see” (s, e, e), where <sil> between tokens is optional and I-x can be skipped for very short tokens:
    • Repeated tokens are separated naturally by the E-e → B-e transition, so no blank is needed between them.
  • Factored output head
    • A joint output layer would triple the vocabulary from V+1 to 3V+1.
    • Instead, a token head (V tokens + <sil>) and a phase head (B, I, E) are trained together, with P(token, phase) ≈ P(token) × P(phase). <sil> is masked out of the phase head.
    • This saves 2V−3 output units, at the cost of an independence approximation.
  • Since the B and E states give the onset and offset directly, BIE-CTC could also help with:
    • streaming (the E state as an emission signal)
    • compressing CTC outputs for speech LLMs
    • duration modeling (the length of the I state as a differentiable duration)

Open Questions

  • Minimum duration: each token needs at least 2 frames (B + E), compared to 1 frame in CTC. This may hurt fast speech, so should a token be allowed to take a single frame?
  • The factored head assumes that the token and the phase are independent, which is only an approximation.
  • The forward-backward algorithm must be rewritten. The standard CTC trellis has 2L+1 states, while BIE-CTC has 4L+1, so the existing CTC losses in ESPnet or k2 cannot be used as is.
  • Chorowski et al. found that frame-level normalization with context-dependent CTC symbols overfits and needed utterance-level normalization [paper]. The same may apply to the larger BIE output space.

Experiment Plan

  • Main claim: more accurate forced alignment
  • Baselines: (1) standard CTC, (2) CTC with label priors, which BIE-CTC must beat
  • Ablations: factored vs. joint output head, mandatory vs. optional I state
  • Data: LibriSpeech for training and WER, TIMIT for alignment
  • Metric: phoneme boundary error (PBE, ms)