Overview

Tacotron is an end-to-end text-to-speech (TTS) synthesis system developed by Google Research. It directly converts graphemes or phonemes into raw audio waveforms using a sequence‑to‑sequence architecture with attention, eliminating the need for traditional pipeline components such as linguistic feature extraction and acoustic model concatenation. The original Tacotron (2017) was followed by Tacotron 2 (2018), which incorporated a WaveNet vocoder for higher fidelity. The system has had significant impact on neural TTS research, enabling more natural and expressive synthetic speech.

1.1 Motivation and historical context

Prior to Tacotron, conventional TTS systems relied on a multi‑stage pipeline: text analysis, linguistic feature extraction, acoustic modelling, and waveform generation. Each stage required hand‑crafted features and domain expertise, and the cascaded architecture often introduced artifacts and limited naturalness. Deep learning had already shown promise in acoustic modelling (e.g., WaveNet) and sequence‑to‑sequence learning (e.g., neural machine translation). Tacotron was proposed as a unified neural approach that could learn the entire mapping from text to audio directly from data, reducing engineering complexity and enabling more expressive output.

1.2 Core architecture design

Tacotron employs an encoder‑decoder framework with an attention mechanism. The encoder converts input characters or phonemes into a learned hidden representation. The decoder, conditioned on this representation, generates a sequence of low‑level acoustic features (e.g., mel‑spectrograms) frame by frame. Attention aligns the decoder outputs with the encoder states, allowing the model to handle variable‑length input and output. The final step is a vocoder (initially Griffin‑Lim, later WaveNet) that converts the mel‑spectrograms into raw audio waveforms. The entire system is trained end‑to‑end, optimizing a combination of L1 and L2 losses on the spectrogram frames.

2 Model variants

2.1 Tacotron (original)

The original Tacotron, introduced in a 2017 paper, demonstrated the feasibility of end‑to‑end TTS. It consisted of an encoder, an attention‑based decoder, and a post‑processing network that produced linear‑scale spectrograms, which were then converted to audio via the Griffin‑Lim algorithm.

2.1.1 Encoder‑decoder with attention

The encoder uses a convolutional bank with highway layers and a bidirectional LSTM to capture both local and long‑range context. The decoder is an autoregressive unidirectional LSTM that outputs a sequence of mel‑spectrogram frames. Location‑sensitive attention provides better alignment stability during training. The decoder also has a “stop token” prediction to determine when synthesis is complete.

2.1.2 Griffin‑Lim vocoder

The original Tacotron used the Griffin‑Lim algorithm, a phase reconstruction method that iteratively estimates the phase from a magnitude spectrogram. While fast and differentiable, its synthesized audio suffered from audible artifacts and lacked the richness of a neural vocoder. This limitation motivated the move to a higher‑fidelity vocoder in later versions.

2.2 Tacotron 2

Tacotron 2, published in 2018, refined the architecture and replaced the Griffin‑Lim vocoder with a modified WaveNet, achieving significantly improved audio quality.

2.2.1 Modified encoder‑decoder

The encoder was simplified: it uses a single convolutional bank followed by a bidirectional LSTM, removing the highway layers. The decoder now outputs mel‑spectrograms at a lower bandwidth (80‑band mel) to reduce the computational load on the vocoder. The attention mechanism remained location‑sensitive but was made more robust with teacher forcing and scheduled sampling.

2.2.2 WaveNet vocoder integration

A separately trained WaveNet autoregressive model conditions on the Tacotron 2 mel‑spectrograms to generate raw audio samples. WaveNet’s dilated causal convolutions capture long‑range temporal dependencies, producing natural‑sounding speech with low distortion. The WaveNet vocoder is substantially more computationally expensive than Griffin‑Lim, but the quality gain is considerable.

2.2.3 Performance improvements

Tacotron 2 reduced the mean opinion score (MOS) gap between synthetic and human speech to near‑human levels in many test conditions. The model also improved robustness to punctuation and capitalization, and could handle long sentences more reliably than the original Tacotron.

2.3 Later iterations and forks

2.3.1 Tacotron 2 with GST (Global Style Tokens)

Global Style Tokens (GST) were proposed as a plug‑in module for Tacotron 2 that learns a set of latent “style” embeddings from unlabelled data. By conditioning the decoder on a weighted combination of these tokens, the system gains control over prosody, speaking style, and emotion without requiring explicit labels. This enabled expressive variations such as cheerful or sad speech.

2.3.2 Non‑autoregressive variants (e.g., FastSpeech)

One major limitation of autoregressive Tacotron models is slow inference due to sequential generation. Non‑autoregressive TTS models like FastSpeech (2019) use a feed‑forward transformer and a duration predictor to generate mel‑spectrograms in parallel. While not strictly a direct fork of Tacotron, FastSpeech was heavily inspired by its successes and often uses Tacotron‑style encoders and decoders as baselines for comparison.

3 Training and data

3.1 Training procedure

Training Tacotron models requires paired text–audio corpora. The model is optimized to predict mel‑spectrograms from character or phoneme sequences. Teacher forcing is used during training, while scheduled sampling can be applied to improve inference‑time robustness.

3.1.1 Loss functions (L1, L2, guided attention)

The primary loss is a combination of L1 and L2 distances between the predicted and ground‑truth mel‑spectrograms. For the original Tacotron, a “guided attention” loss was introduced to encourage monotonic alignment, preventing the decoder from repeating or skipping text regions. This is a diagonal prior that penalizes non‑monotonic attention weights.

3.1.2 Data preprocessing and alignment

Audio is resampled to a common rate (e.g., 24 kHz). Mel‑spectrograms are computed using a short‑time Fourier transform with typical parameters (hop length 256, window size 1024, 80‑band mel filterbank). Text is normalized (expanding numbers, abbreviations) and optionally converted to phonemes. During training, the alignment between text and audio is learned implicitly by the attention mechanism; no forced alignment is required.

3.2 Datasets commonly used

3.2.1 LJSpeech

LJSpeech is a single‑speaker dataset consisting of 13,100 short audio clips of a female speaker reading passages from seven non‑fiction books. It is the most widely used dataset for Tacotron training due to its clean recordings, consistent speaker, and public availability.

3.2.2 VCTK, LibriTTS

VCTK contains 44 hours of speech from 108 English speakers with various accents. It is often used for multi‑speaker Tacotron experiments. LibriTTS is derived from audiobooks, offering 585 hours of read speech with multiple speakers and larger variety in prosody. These datasets enable generalization to new voices and more natural variability.

4 Applications and impact

4.1 Commercial products (Google Assistant, etc.)

Tacotron‑based synthesis has been deployed in Google Assistant, Google Maps navigation, and other Google services. The naturalness of Tacotron 2 allowed these products to shift from concatenative or parametric voices to neural voices that sound nearly human, improving user experience.

4.2 Open‑source implementations (e.g., NVIDIA Tacotron 2)

Several open‑source reproductions of Tacotron 2 exist, including NVIDIA’s implementation in PyTorch and the Mozilla TTS project. These have enabled researchers and hobbyists to train custom voices, experiment with modifications, and push forward the state of the art. Pre‑trained models are also available, lowering the barrier to entry.

4.3 Influence on subsequent TTS models (WaveNet, Glow‑TTS, etc.)

Tacotron’s end‑to‑end approach inspired many later models. WaveNet, while originally a standalone vocoder, became the natural partner for Tacotron 2. Glow‑TTS used a flow‑based decoder to produce mel‑spectrograms in parallel, but retained Tacotron‑style text encoders and attention. FastSpeech and FastPitch replaced Tacotron’s autoregressive decoding with duration‑based parallel generation, directly building on its encoder‑attention framework.

5 Limitations and challenges

5.1 Prosody and emotion control

Tacotron models, especially without GST, have limited ability to control pitch, emphasis, or emotional tone. The output prosody is usually a neutral average of the training data. Adding explicit control (e.g., via reference encoders or style tokens) helps but requires careful tuning.

5.2 Robustness to out‑of‑vocabulary words

When encountering rare words, names, or foreign text, Tacotron can produce mispronunciations or unnatural durations. Phonetic inputs or grapheme‑to‑phoneme dictionaries mitigate this, but the model’s robustness remains imperfect.

5.3 Computational cost and latency

Autoregressive generation makes Tacotron slow for real‑time applications, especially when paired with WaveNet. On typical consumer hardware, synthesis speed is below real‑time. Non‑autoregressive variants solve this but may sacrifice some naturalness.

6.1 WaveNet / Parallel WaveNet

WaveNet is an autoregressive neural vocoder that generates raw audio samples conditioned on linguistic features or mel‑spectrograms. It was the first model to produce speech with human‑level naturalness. However, its sequential generation is extremely slow; Parallel WaveNet used a feed‑forward network with density distillation to achieve real‑time synthesis.

6.2 FastSpeech / FastPitch

FastSpeech (2019) is a non‑autoregressive TTS model that uses a feed‑forward transformer and a duration predictor to generate mel‑spectrograms in parallel. FastPitch adds explicit pitch conditioning, improving prosodic control. Both models build on sequence‑to‑sequence ideas from Tacotron while overcoming its inference speed limitations.

6.3 End‑to‑end TTS taxonomy

End‑to‑end TTS systems can be classified into: (1) two‑stage models (acoustic model + vocoder), e.g., Tacotron 2 + WaveNet; (2) single‑stage models that directly output waveform, e.g., WaveTTS, EATS; and (3) fully convolutional or transformer architectures such as NaturalSpeech. Tacotron belongs to the first category and remains a foundational reference in the field.