Audio Encoders and Speech Models

Turning sound into tokens

Audio encoders follow the same pattern as vision encoders: make a 2-D representation, cut it into a sequence, run a transformer. Whisper is a widely used example. It is an encoder-decoder transformer trained on a large amount of labelled speech in many languages. Audio is resampled to 16 kHz, split into 30-second windows, and converted to a log-mel spectrogram; a small convolutional stem downsamples it in time, and the transformer encoder produces one vector per short time step. The decoder then generates text tokens, including special tokens for language, task (transcribe or translate to English) and timestamps. Self-supervised encoders such as wav2vec 2.0 and HuBERT instead learn from unlabelled audio by predicting masked parts and are then fine-tuned. In a multimodal LLM, the audio encoder's output vectors can be projected into the language model's embedding space, exactly as image patch vectors are.

Transcribing with a Whisper pipeline

The Hugging Face pipeline handles resampling, feature extraction and decoding. It downloads the model on first use, so the output is not shown here.

from transformers import pipeline

asr = pipeline("automatic-speech-recognition", model="openai/whisper-small")
result = asr("lecture_clip.wav", return_timestamps=True)
print(result["text"])
for chunk in result["chunks"]:
    print(chunk["timestamp"], chunk["text"])

Long audio needs chunking

Whisper's encoder sees 30 seconds at a time. For a one-hour lecture, the audio must be chunked (the pipeline can do this with chunk_length_s) and timestamps stitched back together. Watch for words cut at chunk boundaries.

त्वरित जाँच: What input representation does Whisper's encoder take?

• Raw 44.1 kHz waveform samples
• A log-mel spectrogram of 16 kHz audio
• Text tokens from a phonetic dictionary
• MP3 compressed frames