Speech Generation and Voice Interfaces

From text to a voice

Text-to-speech (TTS) turns text into audio. Classic neural TTS has two stages: an acoustic model predicts a mel spectrogram from text (handling pronunciation, rhythm and intonation), and a vocoder converts the spectrogram into a waveform. Newer systems often represent audio as discrete tokens from a neural audio codec and generate those tokens with a transformer, much like generating text, then decode them to sound. A voice assistant can be built as a cascade: speech recognition, then an LLM, then TTS. Cascades are modular and easy to debug, but latency adds up across three models and the LLM never hears tone or emotion. End-to-end speech models take audio tokens in and produce audio tokens out, which lowers latency and preserves paralinguistic cues, at the cost of being harder to inspect and control. Voice cloning raises consent issues covered later in this course.

Dubbing a film

A cascade is like dubbing a Tamil film into Hindi: transcribe the dialogue, translate the script, then have a voice artist record it. It works, but timing and emotion can slip at each hand-off. An end-to-end model is like a bilingual actor performing the scene directly.

Latency budget for voice

Conversation feels natural only when replies start quickly. In a cascade, stream everything: partial transcripts into the LLM, LLM tokens into TTS sentence by sentence, and audio chunks to the user before the full answer exists.

त्वरित जाँच: What is a main weakness of a cascaded ASR -> LLM -> TTS voice assistant?

• It cannot produce any audio
• It needs no speech recognition
• It only works in English by design
• Latency adds up and the LLM loses tone and emotion from the speech