# Audio as Waveforms and Spectrograms — Multimodal AI

Source: https://www.skillbyai.com/en/multimodal-ai/mm-audio-as-tensors

> Explain sample rate, framing and spectrograms, and compute a simple spectrogram with NumPy.

## From air pressure to a time-frequency picture

Digital audio is a **waveform**: a 1-D array of amplitude samples taken at a fixed **sample rate**. Speech models commonly use 16,000 samples per second, so one second of audio is 16,000 numbers. Raw samples are long and repetitive, so most models convert them to a **spectrogram**: cut the signal into short overlapping windows (often about 25 ms, moving 10 ms each step), apply a Fourier transform to each window, and keep the magnitude of each frequency. The result is a 2-D array of time by frequency, which looks like an image and can be processed like one. A **log-mel spectrogram** additionally groups frequencies into mel bands that follow how humans perceive pitch, and takes the logarithm so quiet and loud sounds are on a comparable scale. Whisper, for example, takes a log-mel spectrogram as input. Resampling audio to the rate the model expects is mandatory; feeding 44.1 kHz audio to a 16 kHz model changes every frequency it sees.

## A spectrogram of a pure tone

A 440 Hz sine wave at 16 kHz is framed with a 400-sample window and 160-sample hop. The strongest frequency bin lands exactly at 440 Hz.

```python
import numpy as np

sr = 16000                              # samples per second
t = np.arange(sr) / sr                  # 1 second of timestamps
wave = 0.5 * np.sin(2 * np.pi * 440 * t)
print(wave.shape, "samples")

win, hop = 400, 160                     # 25 ms window, 10 ms hop
frames = [wave[i:i + win] * np.hanning(win)
          for i in range(0, len(wave) - win + 1, hop)]
spec = np.abs(np.fft.rfft(np.stack(frames), axis=1))
print(spec.shape, "(frames, frequency bins)")

peak_bin = spec.mean(axis=0).argmax()
print("peak at", peak_bin * sr / win, "Hz")
```

Output:

```
(16000,) samples
(98, 201) (frames, frequency bins)
peak at 440.0 Hz
```

## A music teacher's notation

A waveform is like recording the vibration of a sitar string every instant. A spectrogram is like the notation a teacher writes: which notes sound at each moment. The notation is shorter and far easier to reason about.

**Quiz:** A 10-second clip is sampled at 16 kHz. How many samples does the waveform have?

- [ ] 16,000
- [ ] 1,600
- [x] 160,000
- [ ] 10,000

*Answer:* 160,000. Sample rate times duration: 16,000 samples per second x 10 seconds = 160,000 samples.