Audio as Waveforms and Spectrograms

From air pressure to a time-frequency picture

Digital audio is a waveform: a 1-D array of amplitude samples taken at a fixed sample rate. Speech models commonly use 16,000 samples per second, so one second of audio is 16,000 numbers. Raw samples are long and repetitive, so most models convert them to a spectrogram: cut the signal into short overlapping windows (often about 25 ms, moving 10 ms each step), apply a Fourier transform to each window, and keep the magnitude of each frequency. The result is a 2-D array of time by frequency, which looks like an image and can be processed like one. A log-mel spectrogram additionally groups frequencies into mel bands that follow how humans perceive pitch, and takes the logarithm so quiet and loud sounds are on a comparable scale. Whisper, for example, takes a log-mel spectrogram as input. Resampling audio to the rate the model expects is mandatory; feeding 44.1 kHz audio to a 16 kHz model changes every frequency it sees.

A spectrogram of a pure tone

A 440 Hz sine wave at 16 kHz is framed with a 400-sample window and 160-sample hop. The strongest frequency bin lands exactly at 440 Hz.

import numpy as np

sr = 16000                              # samples per second
t = np.arange(sr) / sr                  # 1 second of timestamps
wave = 0.5 * np.sin(2 * np.pi * 440 * t)
print(wave.shape, "samples")

win, hop = 400, 160                     # 25 ms window, 10 ms hop
frames = [wave[i:i + win] * np.hanning(win)
          for i in range(0, len(wave) - win + 1, hop)]
spec = np.abs(np.fft.rfft(np.stack(frames), axis=1))
print(spec.shape, "(frames, frequency bins)")

peak_bin = spec.mean(axis=0).argmax()
print("peak at", peak_bin * sr / win, "Hz")

Output:

(16000,) samples
(98, 201) (frames, frequency bins)
peak at 440.0 Hz

A music teacher's notation

A waveform is like recording the vibration of a sitar string every instant. A spectrogram is like the notation a teacher writes: which notes sound at each moment. The notation is shorter and far easier to reason about.

Quick check: A 10-second clip is sampled at 16 kHz. How many samples does the waveform have?

• 16,000
• 1,600
• 160,000
• 10,000