Data Contamination in Speech Modeling (2025–)


Motivation

  • As the training data of speech foundation models grows to hundreds of thousands of hours crawled from the web, some of the test data inevitably leaks into the training data.
    • For example, OWSM v4 adds YODAS, a web-crawled corpus, to its training data [paper]
  • The same problem has been widely studied for text LLMs.
    • GPT-3 measured the overlap between its training data and the benchmarks with 13-gram matching [paper]
    • Jiang et al. show that n-gram-based definitions miss paraphrased test data and flag documents that are not really contaminated [paper]
  • In speech, Tseng et al. found that large parts of the LibriSpeech and Common Voice test sets appear in public LLM pre-training data, such as Project Gutenberg and Wikipedia [paper]
    • Speech recognizers built on contaminated LLMs showed only small differences in WER, but assigned much higher probabilities to the test transcriptions they had seen.

Contamination is Not Necessarily Bad

  • Seeing test data in training does not always mean that a model exploits it.
    • Magar and Schwartz separate memorization from exploitation: in some cases models use contaminated examples to perform better, and in others they memorize them without any benefit [paper]
    • Jiang et al. find that the effect depends on how often the contaminated data is repeated: performance first improves, then drops (sometimes below the baseline) after about 10 repetitions [paper]
  • So, rather than removing every overlap, we need to clearly understand how each type of contamination affects benchmark results.

Goals

  1. Define data contamination for speech modeling.
  2. Analyze the correlation between data contamination and benchmark performance.

Defining Data Contamination

  • Unlike text, speech data can overlap with the test data in its transcript, its audio, or both.
  • Text contamination: the transcript of a test utterance appears in the training data
    • Full text: the normalized transcript matches exactly.
    • N-gram overlap: a part of the transcript overlaps, measured by the ratio of shared n-grams.
  • Audio contamination: the audio of the training data shares features with the test utterance
    • Speaker: the same speaker appears in both the training and the test data. This can be measured with speaker embeddings, such as ECAPA-TDNN [paper]