TokSing: Singing Voice Synthesis based on Discrete Tokens

Overview

• This paper introduces TokSing, a novel singing voice synthesis model that generates singing audio from discrete tokens.
• TokSing uses a transformer-based architecture to generate singing voice from text, melody, and other input features.
• The authors demonstrate that TokSing outperforms existing singing voice synthesis models on objective and subjective evaluation metrics.

Plain English Explanation

TokSing is a new system that can generate singing voices from text, melody, and other information. It uses a type of neural network called a transformer to convert these inputs into realistic-sounding singing. This is an important advancement in the field of singing voice synthesis, which aims to create artificial singing voices that sound natural and human-like.

The key innovation in TokSing is its use of "discrete tokens" - small, distinct units of information that represent the different aspects of singing, like pitch, rhythm, and timbre. By modeling singing at this more granular level, TokSing is able to generate more nuanced and expressive singing voices compared to previous approaches, which tended to produce more robotic-sounding results.

Overall, the authors show that TokSing outperforms existing singing voice synthesis systems on both objective metrics (like how closely the generated audio matches the target) and subjective evaluations (how natural and musical the singing sounds to human listeners). This represents an important step forward in the quest to create high-quality, data-efficient singing voice synthesis systems.

Technical Explanation

TokSing uses a transformer-based architecture to generate singing voice from text, melody, and other input features. The core of the model is a vector quantized variational autoencoder (VQ-VAE) that encodes the input features into a sequence of discrete tokens. These tokens represent the fundamental units of singing, such as pitch, duration, and timbre.

The encoder network maps the input features (text, melody, etc.) into a high-dimensional latent space, which is then quantized into a sequence of discrete tokens by the VQ-VAE. The decoder network then takes this token sequence and generates the corresponding singing voice audio. This two-stage process allows the model to learn a more granular and expressive representation of singing compared to previous end-to-end approaches.

The authors also incorporate diverse semantic-based audio pretraining to initialize the encoder and improve the model's generalization. Experiments on public singing voice datasets show that TokSing outperforms state-of-the-art baselines in both objective and subjective evaluations of singing quality.

Critical Analysis

The authors acknowledge several limitations of the current TokSing model. First, the training dataset used is relatively small, consisting of only about 2 hours of singing data. Scaling up the singing voice data used to train the model could lead to further improvements in synthesis quality.

Additionally, the authors note that TokSing is currently limited to generating singing in a single language (English). Extending the model to handle multilingual singing synthesis is an important area for future work. The authors also suggest that incorporating more diverse musical styles and genres into the training data could enhance the model's expressive capabilities.

Overall, TokSing represents a promising step forward in singing voice synthesis research. By leveraging discrete tokens and semantic-based pretraining, the model is able to generate more natural and musical singing voices than previous approaches. However, further research is needed to scale up the data, extend the model to new languages and genres, and fully realize the potential of this discrete token-based synthesis paradigm.

Conclusion

The TokSing paper introduces a novel singing voice synthesis model that generates high-quality singing audio from text, melody, and other input features. The key innovation is the use of a vector quantized variational autoencoder to encode the input into a sequence of discrete tokens, which allows the model to learn a more granular and expressive representation of singing.

Experiments show that TokSing outperforms existing singing voice synthesis systems on both objective and subjective evaluation metrics. This represents an important advancement in the field, bringing us closer to the goal of data-efficient, high-quality singing voice synthesis that can be widely deployed in various applications.

While the current TokSing model has some limitations in terms of dataset size and language support, the authors have outlined promising directions for future research. Scaling up the training data, extending to multilingual singing, and incorporating more diverse musical styles could further enhance the model's capabilities. Overall, this paper makes a valuable contribution to the ongoing efforts to create advanced singing voice synthesis systems that can produce natural, expressive, and human-like singing.