Cross-domain Neural Pitch and Periodicity Estimation

Overview

• Pitch is a fundamental aspect of how we perceive audio signals.
• Pitch contours are commonly used to analyze speech and music signals and as input features for many audio tasks.
• This paper describes techniques to improve the accuracy of neural pitch and periodicity estimators, achieving state-of-the-art performance on both speech and music.
• A novel entropy-based method is introduced for extracting periodicity and voiced-unvoiced classifications from statistical inference-based pitch estimators.
• The paper shows how to train a neural pitch estimator to handle both speech and music data without performance degradation.
• The estimator implementations run extremely fast, approaching the speed of state-of-the-art DSP-based pitch estimators.
• An open-source Python module called Pitch-Estimating Neural Networks (penn) is released for training, evaluating, and performing inference with pitch- and periodicity-estimating neural networks.

Plain English Explanation

Pitch is a fundamental aspect of how we perceive sound. It refers to whether a sound is high or low. Pitch contours are commonly used to analyze speech and music signals, and they are also used as input features for many audio-related tasks, such as music transcription, singing voice synthesis, and prosody editing.

In this paper, the researchers describe techniques they developed to improve the accuracy of widely-used neural pitch and periodicity estimators. Periodicity refers to the regularity of a sound wave. By using these techniques, the researchers were able to achieve state-of-the-art performance on pitch estimation for both speech and music.

The researchers also introduced a novel method for extracting periodicity and whether a sound is voiced (produced by vibrating vocal cords) or unvoiced from statistical inference-based pitch estimators, such as neural networks. Additionally, they showed how to train a neural pitch estimator to handle both speech and music data without any loss in performance.

Importantly, the researchers' estimator implementations run extremely fast, approaching the speed of specialized hardware-based pitch estimators, or even hundreds of times faster on a GPU. This means the pitch estimation can be done very quickly, which is important for real-time applications.

The researchers have released all of their code and models as an open-source Python module called Pitch-Estimating Neural Networks (penn), which allows others to train, evaluate, and use their pitch and periodicity estimating neural networks.

Technical Explanation

The paper focuses on improving the accuracy of neural pitch and periodicity estimators for both speech and music signals. The researchers developed several key techniques:

1. Improved Pitch Estimation Accuracy: The researchers applied various enhancements to widely-used neural pitch estimators to achieve state-of-the-art performance on both speech and music data. This included techniques like improved loss functions and data augmentation.

2. Novel Periodicity Extraction Method: The researchers introduced a novel entropy-based method for extracting periodicity and per-frame voiced-unvoiced classifications from statistical inference-based pitch estimators, such as neural networks. This allows for more accurate and robust periodicity estimation.

3. Cross-Domain Pitch Estimation: The researchers showed how to train a single neural pitch estimator to handle both speech and music data without any performance degradation. This "cross-domain" estimation capability is important for real-world applications.

4. Extremely Fast Inference: The researchers' estimator implementations run extremely fast, achieving 11.2x real-time performance on a high-end CPU and 408x real-time performance on a GPU. This approaches the speed of specialized DSP-based pitch estimators, making the neural-network-based approach viable for real-time applications.

The researchers evaluated their techniques on standard speech and music pitch estimation benchmarks, demonstrating significant improvements over previous state-of-the-art methods. They also released their code and models as the open-source Pitch-Estimating Neural Networks (penn) library, allowing others to build upon their work.

Critical Analysis

The paper presents a thorough and well-designed set of techniques for improving neural-network-based pitch and periodicity estimation. The novel entropy-based periodicity extraction method is a particularly interesting contribution, as it allows for more robust and accurate periodicity estimation compared to previous approaches.

One potential limitation is that the paper does not provide a deep analysis of the failure cases or limitations of their proposed methods. While the researchers demonstrate state-of-the-art performance, it would be valuable to understand the specific scenarios where the estimators may struggle or produce inaccurate results.

Additionally, the paper does not explore the implications of the fast inference speeds for real-world applications. It would be informative to see examples of how the researchers' techniques could be leveraged in practical audio processing systems, such as robust speech separation models for similar-pitch speakers or automatic equalization of individual instrument tracks.

Overall, the paper presents a significant advancement in the field of neural pitch and periodicity estimation, with the open-sourcing of the penn library being a valuable contribution to the research community. Further exploration of the practical applications and limitations of the techniques could lead to even greater impact.

Conclusion

This paper introduces a set of innovative techniques to improve the accuracy and performance of neural pitch and periodicity estimators for both speech and music signals. The researchers achieved state-of-the-art results on standard benchmarks and developed a novel entropy-based method for extracting periodicity and voiced-unvoiced classifications. Importantly, the researchers' estimator implementations run extremely fast, approaching the speed of specialized hardware-based pitch estimators.

The open-sourcing of the penn library allows others to build upon this work and leverage the researchers' pitch and periodicity estimation capabilities in a wide range of audio processing applications, from music transcription to singing voice synthesis and beyond. As pitch and periodicity are fundamental to our perception of sound, the techniques described in this paper have the potential to enable significant advancements in numerous audio-related fields.