# Audio Encoders and Speech Models — Multimodal AI

Source: https://www.skillbyai.com/en/multimodal-ai/mm-audio-encoders

> Describe how speech encoders such as Whisper process log-mel spectrograms and where they fit in a multimodal system.

## Turning sound into tokens

Audio encoders follow the same pattern as vision encoders: make a 2-D representation, cut it into a sequence, run a transformer. **Whisper** is a widely used example. It is an encoder-decoder transformer trained on a large amount of labelled speech in many languages. Audio is resampled to 16 kHz, split into 30-second windows, and converted to a log-mel spectrogram; a small convolutional stem downsamples it in time, and the transformer encoder produces one vector per short time step. The decoder then generates text tokens, including special tokens for language, task (transcribe or translate to English) and timestamps. Self-supervised encoders such as wav2vec 2.0 and HuBERT instead learn from unlabelled audio by predicting masked parts and are then fine-tuned. In a multimodal LLM, the audio encoder's output vectors can be projected into the language model's embedding space, exactly as image patch vectors are.

## Transcribing with a Whisper pipeline

The Hugging Face pipeline handles resampling, feature extraction and decoding. It downloads the model on first use, so the output is not shown here.

```python
from transformers import pipeline

asr = pipeline("automatic-speech-recognition", model="openai/whisper-small")
result = asr("lecture_clip.wav", return_timestamps=True)
print(result["text"])
for chunk in result["chunks"]:
    print(chunk["timestamp"], chunk["text"])
```

## Long audio needs chunking

Whisper's encoder sees 30 seconds at a time. For a one-hour lecture, the audio must be chunked (the pipeline can do this with `chunk_length_s`) and timestamps stitched back together. Watch for words cut at chunk boundaries.

**Quiz:** What input representation does Whisper's encoder take?

- [ ] Raw 44.1 kHz waveform samples
- [x] A log-mel spectrogram of 16 kHz audio
- [ ] Text tokens from a phonetic dictionary
- [ ] MP3 compressed frames

*Answer:* A log-mel spectrogram of 16 kHz audio. Whisper resamples to 16 kHz and computes a log-mel spectrogram, processed in 30-second windows.