In text-to-speech (TTS) synthesis, diffusion models have achieved promising generation quality. However, because of the pre-defined data-to-noise diffusion process, their prior distribution is restricted to a noisy representation, which provides little information of the generation target. In this work, we present a novel TTS system, Bridge-TTS, making the first attempt to substitute the noisy Gaussian prior in established diffusion-based TTS methods with a clean and deterministic one, which provides strong structural information of the target. Specifically, we leverage the latent representation obtained from text input as our prior, and build a fully tractable Schrodinger bridge between it and the ground-truth mel-spectrogram, leading to a data-to-data process. Moreover, the tractability and flexibility of our formulation allow us to empirically study the design spaces such as noise schedules, as well as to develop stochastic and deterministic samplers. Experimental results on the LJ-Speech dataset illustrate the effectiveness of our method in terms of both synthesis quality and sampling efficiency, significantly outperforming our diffusion counterpart Grad-TTS in 50-step/1000-step synthesis and strong fast TTS models in few-step scenarios.
(1) Theoretically, we propose a fully tractable Schrodinger Bridge between paired data, tailored for the TTS task. We also propose bridge SDE/ODE for inference, and develop samplers with exponential integrators.
(2) Empirically, we study the design spaces of noise schedules, training objectives and sampling schemes.
We conduct subjective tests on the open platform Amazon Mechanical Turk, with 20 samples and 25 subjects each model.
Bridge-TTS achieves subjective quality close to the ground truth with only 50 steps, significantly outperforming previous diffusion-based methods.
Ground-Truth Recording
GTMel + Vocoder
VITS
Bridge-TTS-VP-1000 (Ours)
Bridge-TTS-gmax-1000 (Ours)
Grad-TTS-1000
Bridge-TTS-VP-50 (Ours)
Bridge-TTS-gmax-50 (Ours)
Grad-TTS-50
DiffSinger-71
ResGrad-50
FastSpeech 2
LJ001-0009: Printing, then, for our purpose, may be considered as the art of making books by means of movable types.
Ground-Truth Recording
GTMel + Vocoder
VITS
Bridge-TTS-VP-1000 (Ours)
Bridge-TTS-gmax-1000 (Ours)
Grad-TTS-1000
Bridge-TTS-VP-50 (Ours)
Bridge-TTS-gmax-50 (Ours)
Grad-TTS-50
DiffSinger-71
ResGrad-50
FastSpeech 2
LJ001-0014: And it was a matter of course that in the Middle Ages, when the craftsmen took care that beautiful form should always be a part of their productions whatever they were
Ground-Truth Recording
GTMel + Vocoder
VITS
Bridge-TTS-VP-1000 (Ours)
Bridge-TTS-gmax-1000 (Ours)
Grad-TTS-1000
Bridge-TTS-VP-50 (Ours)
Bridge-TTS-gmax-50 (Ours)
Grad-TTS-50
DiffSinger-71
ResGrad-50
FastSpeech 2
LJ001-0010: Now, as all books not primarily intended as picture-books consist principally of types composed to form letterpress.
Ground-Truth Recording
GTMel + Vocoder
VITS
Bridge-TTS-VP-1000 (Ours)
Bridge-TTS-gmax-1000 (Ours)
Grad-TTS-1000
Bridge-TTS-VP-50 (Ours)
Bridge-TTS-gmax-50 (Ours)
Grad-TTS-50
DiffSinger-71
ResGrad-50
FastSpeech 2
LJ001-0001: Printing, in the only sense with which we are at present concerned, differs from most if not from all the arts and crafts represented in the Exhibition.
Bridge-TTS outperforms previous methods with only 2 steps.
Ground-Truth Recording
GTMel + Vocoder
FastSpeech 2
Bridge-TTS-gmax-4 (Ours)
Fast-GradTTS-4
Grad-TTS-4
ResGrad-4
DiffGAN-TTS-4
ProDiff-2
Bridge-TTS-gmax-2 (Ours)
CoMoSpeech-2
CoMoSpeech-1
LJ001-0009: Printing, then, for our purpose, may be considered as the art of making books by means of movable types.
Ground-Truth Recording
GTMel + Vocoder
FastSpeech 2
Bridge-TTS-gmax-4 (Ours)
Fast-GradTTS-4
Grad-TTS-4
ResGrad-4
DiffGAN-TTS-4
ProDiff-2
Bridge-TTS-gmax-2 (Ours)
CoMoSpeech-2
CoMoSpeech-1
LJ001-0014: And it was a matter of course that in the Middle Ages, when the craftsmen took care that beautiful form should always be a part of their productions whatever they were.
Ground-Truth Recording
GTMel + Vocoder
FastSpeech 2
Bridge-TTS-gmax-4 (Ours)
Fast-GradTTS-4
Grad-TTS-4
ResGrad-4
DiffGAN-TTS-4
ProDiff-2
Bridge-TTS-gmax-2 (Ours)
CoMoSpeech-2
CoMoSpeech-1
LJ001-0010: Now, as all books not primarily intended as picture-books consist principally of types composed to form letterpress.
@misc{chen2023schrodinger,
title={Schrodinger Bridges Beat Diffusion Models on Text-to-Speech Synthesis},
author={Zehua Chen and Guande He and Kaiwen Zheng and Xu Tan and Jun Zhu},
year={2023},
}