Few-/Zero-shot Speech-Text Alignment via Isomorphism

Few-/Zero-shot Speech-Text Alignment via Isomorphism

Motivation

• Models trained separately on different modalities may still learn similar geometries.
  • The Platonic Representation Hypothesis argues that as vision and language models grow, they measure the distance between data points in more and more similar ways [paper]
• If the geometries are isomorphic, the two representation spaces can be aligned with minimal or no paired data.
  • Schnaus et al. align independently trained vision and language models without any pairs [paper]
    
    _{Figure 1 from Schnaus et al. The embeddings of DINOv2 (images) and Qwen3-8B (text) share a similar geometry, which allows alignment without paired data.}
    • Coarse initialization: both embedding sets are clustered with k-means, and the clusters are matched so that the geometric similarity (CKA) between the two sets is maximized.
    • Refinement (Wasserstein Procrustes): it alternates between matching samples across the two sets and fitting a single orthogonal map between them.
    • A geometric similarity score (CKA) between the two spaces predicts well whether unpaired alignment will work.
    • With no pairs, the alignment already enables zero-shot classification and text-to-image generation, and with only a few pairs it clearly outperforms methods that rely on pairs.
• Similar ideas existed for speech before: wav2vec-U learns a speech recognizer without any paired data by matching the distributions of segmented speech representations and phonemized text with adversarial training [paper]

Challenge in Speech

• The geometric search is harder for speech, because speech representations are not trained to have a semantic geometry.
  • Most speech SSL models, such as HuBERT, are trained with a masked prediction (MLM-style) loss [paper]
  • Even wav2vec 2.0, which uses a contrastive loss, contrasts masked frames against other frames from the same utterance, not against other utterances or texts [paper]
• As a result, their representations are frame-level and mostly encode acoustic and phonetic information, rather than an utterance-level embedding whose distances reflect meaning, as in image encoders or sentence encoders.