Abstract

Improving text representation has attracted much attention to achieve expressive text-to-speech (TTS). However, existing works only implicitly learn the prosody with masked token tasks, which leads to low training efficiency and difficulty in prosody modeling. We propose CLAPSpeech, a cross-modal contrastive pre-training framework that learns from the prosody variance of the same text token under different contexts. Specifically, 1) with the design of a text encoder and a prosody encoder, we encourage the model to connect the text context with its corresponding prosody pattern in the joint multi-modal space; 2) we introduce a multi-scale pre-training pipeline to capture prosody patterns in multiple levels. 3) we show how to incorporate CLAPSpeech into existing TTS models for better prosody. Experiments on three datasets not only show that CLAPSpeech could improve the prosody prediction for existing TTS methods, but also demonstrate its generalization ability to adapt to multiple languages and multi-speaker text-to-speech. We also deeply analyze the principle behind the performance of CLAPSpeech. Ablation studies demonstrate the necessity of each component in CLAPSpeech.

Audio Samples

We provide the audio samples generated by the TTS systems of LJSpeech (a single-speaker English dataset), Biaobei (a single-speaker Chinese dataset), and LibriTTS (a multi-speaker English dataset). We use FastSpeech 2 as the prediction-based (PB) TTS baseline, and use PortaSpeech as the variation-based (VB) TTS baseline.

LJSpeech (English, single-speaker)

  1. "The Chronicles of Newgate, Volume 2. By Arthur Griffiths. Section 5: Newgate down to 1818. Part 2"
    2. GT PB baseline PB + BERT PB + A3T PB + CLAPSpeech
    wav
    GT (voc.) VB baseline VB + BERT VB + A3T VB + CLAPSpeech
    wav

  2. "Prisoners were committed to it quite without reference to its capacity."
    3. GT PB baseline PB + BERT PB + A3T PB + CLAPSpeech
    wav
    GT (voc.) VB baseline VB + BERT VB + A3T VB + CLAPSpeech
    wav

  3. "No steps taken to reduce the number of committals, and the governor was obliged to utilize the chapel as a day and night room."
    4. GT PB baseline PB + BERT PB + A3T PB + CLAPSpeech
    wav
    GT (voc.) VB baseline VB + BERT VB + A3T VB + CLAPSpeech
    wav

  4. "And debtors might practically have as much as they liked, if they could only pay for it."
    5. GT PB baseline PB + BERT PB + A3T PB + CLAPSpeech
    wav
    GT (voc.) VB baseline VB + BERT VB + A3T VB + CLAPSpeech
    wav

  5. "For this and other acts of misconduct there was the discipline of the refractory ward, or "strong room" on the debtors' side."
    6. GT PB baseline PB + BERT PB + A3T PB + CLAPSpeech
    wav
    GT (voc.) VB baseline VB + BERT VB + A3T VB + CLAPSpeech
    wav

BiaoBei (Mandarin, single-speaker)

  1. 信xìn 纸zhǐ 一yī 承chéng 以yǐ 上shàng 风fēng 格gé , 一yī 副fù 年nián 逾yú 古gǔ 稀xī 的de 残cán 败bài 样yàng 。
    2. GT PB baseline PB + BERT PB + A3T PB + CLAPSpeech
    wav
    GT (voc.) VB baseline VB + BERT VB + A3T VB + CLAPSpeech
    wav

  2. 孔kǒng 明míng 灯dēng 相xiāng 传chuán 由yóu 诸zhū 葛gě 孔kǒng 明míng 发fā 明míng 。
    3. GT PB baseline PB + BERT PB + A3T PB + CLAPSpeech
    wav
    GT (voc.) VB baseline VB + BERT VB + A3T VB + CLAPSpeech
    wav

  3. 加jiā 快kuài 拱gǒng 北běi 口kǒu 岸àn 改gǎi 扩kuò 建jiàn 和hé 横héng 琴qín 口kǒu 岸àn 客kè 货huò 车chē 通tōng 道dào 重chóng 建jiàn 。
    4. GT PB baseline PB + BERT PB + A3T PB + CLAPSpeech
    wav
    GT (voc.) VB baseline VB + BERT VB + A3T VB + CLAPSpeech
    wav

LibriTTS (English, multi-speaker)

  1. "If at first you don't succeed, try try again."
    2. GT PB baseline PB + BERT PB + A3T PB + CLAPSpeech
    wav
    GT (voc.) VB baseline VB + BERT VB + A3T VB + CLAPSpeech
    wav
  2. "He was, however, driven to resolve that he must go direct to Sophie, as otherwise, he could find no means of doing as he had promised."
    3. GT PB baseline PB + BERT PB + A3T PB + CLAPSpeech
    wav
    GT (voc.) VB baseline VB + BERT VB + A3T VB + CLAPSpeech
    wav
  3. "The stove had a lot of little openings. In one he would put an egg, in another some coffee, and another a piece of meat, and in the fourth some water."
    4. GT PB baseline PB + BERT PB + A3T PB + CLAPSpeech
    wav
    GT (voc.) VB baseline VB + BERT VB + A3T VB + CLAPSpeech
    wav

Case Study: Fine-Grained Prosody Transfer (Section 4.3.2)

We use the text encoder of CLAPSpeech to extract the text prosody encoding of s7 and s8, then replace the text token encoding of “higher” in s8 with that in s7. The text of s7/s8 are shown in the table below.

sentence text
s7 (reference) Higher up could be seen some chinamen.
s8 (source/transfered) And create higher bodies for themselves!

We provide the audio and mel-spectrogram of the reference, source, and transferred sample in the following table. We can hear that by editing the text encoding of “higher” in the source sample, the prosody pattern of “higher” in the reference (the pitch contours in reference remain flat in the early stage and then rise in the late stage) has been successfully transfered into the transfered sample. The successful manipulation of the local prosody proves that our CLAPSpeech extract prosody representation effectively influences the prosody prediction of the TTS system.

    Reference Source Transfered
    wav
    mel reference source transfered