Parallel Synthesis for Autoregressive Speech Generation

Overview

• Autoregressive speech synthesis models can generate highly natural speech, but the iterative generation process makes them slow and inefficient.
• To address this issue, the paper proposes a new autoregressive approach called Frequency-wise Autoregressive (FAR) and Bit-wise Autoregressive (BAR) Speech Synthesis.
• The proposed model splits the speech signal into frequency subbands or quantized bits, and generates each subband or bit iteratively, allowing for parallel computation and faster synthesis.
• The model also employs a post-filter to sample high-quality audio signals from the output distributions.

Plain English Explanation

The paper presents a novel approach to speech synthesis that aims to be more efficient than traditional autoregressive models. Autoregressive models, which generate speech one sample at a time, can produce very natural-sounding speech, but they are slow because the synthesis time is proportional to the length of the speech utterance.

To address this issue, the proposed model takes a different approach. Instead of generating the speech samples one by one in the time domain, the model first splits the speech signal into different frequency subbands. It then generates each subband iteratively, conditioned on the previously generated subband. This Frequency-wise Autoregressive (FAR) approach allows the model to compute in parallel, rather than sequentially, making the synthesis process much faster.

The model also uses a Bit-wise Autoregressive (BAR) approach, where it generates the speech signal one quantized bit at a time. This further improves the efficiency of the synthesis process.

Finally, the model employs a post-filter to generate high-quality audio signals from the output distributions, ensuring that the synthesized speech sounds natural and human-like.

The experimental results show that the proposed model can synthesize speech faster than real-time, without the need for GPU acceleration. It also outperforms both traditional autoregressive and non-autoregressive models in terms of subjective speech quality, while maintaining good generalization to unseen speakers and 44 kHz speech.

Technical Explanation

The paper presents a new approach to autoregressive speech synthesis, called Frequency-wise Autoregressive (FAR) and Bit-wise Autoregressive (BAR) Speech Synthesis. Unlike traditional autoregressive models, which generate speech samples one by one in the time domain, the proposed model generates speech in the frequency domain or the quantized bit domain.

In the FAR approach, the speech signal is first split into different frequency subbands. The model then generates each subband iteratively, conditioned on the previously generated subband. This allows for parallel computation, as the generation of each subband is independent of the others. The full-band speech is then reconstructed using the generated subbands and a synthesis filter bank.

The BAR approach is similar, but instead of generating subbands, the model generates the speech signal one quantized bit at a time. Again, this allows for parallel computation, as the generation of each bit is independent of the others.

By redesigning the autoregressive method to work in the frequency or bit domains, rather than the time domain, the number of iterations required is no longer proportional to the length of the speech utterance, but rather to the number of subbands or bits. This significantly increases the inference efficiency of the model.

The paper also introduces a post-filter that samples high-quality audio signals from the output distributions of the FAR and BAR models. The training objective of this post-filter is designed to take into account the characteristics of the proposed autoregressive methods.

The experimental results show that the proposed model is able to synthesize speech faster than real-time, without the need for GPU acceleration. Compared to both autoregressive and non-autoregressive baseline models, the proposed model achieves better Mean Opinion Scores (MOS) and demonstrates good generalization to unseen speakers and 44 kHz speech.

Critical Analysis

The paper presents a novel and interesting approach to improving the efficiency of autoregressive speech synthesis models. By generating speech in the frequency domain or the quantized bit domain, rather than the time domain, the proposed model is able to achieve significant speedups in the synthesis process.

One potential limitation of the approach is that it may not capture certain temporal dependencies in the speech signal, which could impact the naturalness of the synthesized speech. The authors acknowledge this and suggest that incorporating some temporal context into the model could be an area for future research.

Additionally, the paper does not provide a detailed analysis of the computational complexity of the proposed model, which would be useful for understanding the practical implications of the approach. It would also be interesting to see how the model performs on larger and more diverse speech datasets, as the current evaluation is limited to a single dataset.

Overall, the Frequency-wise Autoregressive (FAR) and Bit-wise Autoregressive (BAR) Speech Synthesis approach presented in the paper is a promising step towards more efficient and high-quality speech synthesis. The novel design choices and the promising results suggest that this could be a valuable contribution to the field of text-to-speech and singing voice synthesis.

Conclusion

The paper proposes a new approach to autoregressive speech synthesis, called Frequency-wise Autoregessive (FAR) and Bit-wise Autoregressive (BAR) Speech Synthesis, that aims to address the inefficiency of traditional autoregressive models. By generating speech in the frequency or quantized bit domains, rather than the time domain, the proposed model is able to achieve significant speedups in the synthesis process without compromising the quality of the generated speech.

The experimental results are promising, showing that the model can synthesize speech faster than real-time and outperform both autoregressive and non-autoregressive baseline models in terms of subjective speech quality. This approach represents an important step towards more efficient and high-quality speech synthesis and could have valuable applications in a wide range of domains, from text-to-speech to singing voice synthesis.