A Multitask Training Approach to Enhance Whisper with Contextual Biasing and Open-Vocabulary Keyword Spotting

Overview

• Recognizing rare named entities like personal names and technical terms is challenging for automatic speech recognition (ASR) systems, especially when they are not common in the training data.
• The paper introduces KWS-Whisper, a novel ASR system that leverages the Whisper model and performs open-vocabulary keyword spotting (OV-KWS) to recognize user-defined named entities, which then serve as prompts for the Whisper decoder.
• The authors propose a multitask training approach to optimize the model, learning both OV-KWS and contextual-ASR tasks.
• The system is evaluated on Chinese Aishell hot word subsets and internal code-switching test sets, showing significant improvements in entity recall compared to the original Whisper model.
• The OV-KWS component can also be used as a plug-and-play module to enhance ASR error correction methods and frozen Whisper models.

Plain English Explanation

Automatic speech recognition (ASR) systems, like the ones used in voice assistants, often struggle to accurately recognize rare or specialized terms, such as people's names or technical jargon. This is because these systems are typically trained on more common speech patterns and vocabulary, so they don't perform as well when encountering less frequent words.

To address this issue, the researchers developed a new ASR system called KWS-Whisper. This system builds on the Whisper ASR model, but adds a keyword spotting component that can recognize user-defined named entities, such as people's names or specialized terms. These recognized entities then serve as cues to help the Whisper model better understand and transcribe the full speech.

To train the system, the researchers used a multitask learning approach, where the model was tasked with both identifying the keywords and transcribing the overall speech. This helped the system learn to better leverage the keyword information to improve the accuracy of the full speech transcription.

When tested on datasets with rare named entities, the KWS-Whisper system significantly outperformed the original Whisper model in correctly recognizing those specialized terms. The researchers also found that the keyword spotting component could be easily integrated with other ASR error correction methods or even used to enhance pre-trained Whisper models.

Overall, this research demonstrates a promising approach to improving the performance of ASR systems on rare and specialized vocabulary, which could have important applications in fields like zero-shot language understanding and bridging the performance gap between human and machine transcription.

Technical Explanation

The paper introduces a novel automatic speech recognition (ASR) system called Keyword Spotting enhanced Whisper (KWS-Whisper), which leverages the Whisper model and performs open-vocabulary keyword spotting (OV-KWS) to recognize user-defined named entities.

The key components of the KWS-Whisper system are:

1. Whisper Encoder: The Whisper encoder is used to generate hidden representations of the input speech.
2. Open-Vocabulary Keyword Spotting (OV-KWS): The OV-KWS module performs keyword spotting on the Whisper encoder's hidden states to detect user-defined named entities.
3. Whisper Decoder: The recognized named entities from the OV-KWS module are used as prompts to guide the Whisper decoder in transcribing the full speech.

To optimize the KWS-Whisper model, the authors propose a multitask training approach that jointly learns the OV-KWS and contextual-ASR tasks. This helps the model better leverage the keyword information to improve the overall speech transcription accuracy.

The system is evaluated on two test sets:

1. Chinese Aishell hot word subsets: This dataset contains rare named entities, such as personal names and technical terminologies.
2. Internal code-switching test sets: These datasets include mixed-language speech with code-switching between Chinese and English.

The results show that the KWS-Whisper system significantly outperforms the original Whisper model in terms of entity recall on these challenging datasets. Additionally, the authors demonstrate that the OV-KWS component can be used as a plug-and-play module to enhance ASR error correction methods and even to improve the performance of frozen Whisper models.

Critical Analysis

The paper presents a compelling approach to improving automatic speech recognition (ASR) systems' ability to handle rare named entities, which is an important and practical challenge in real-world applications. The authors' use of a multitask learning strategy to jointly optimize the keyword spotting and speech transcription tasks is a well-designed solution that leverages the complementary strengths of these two components.

One potential limitation of the study is the relatively narrow evaluation scope, which focuses primarily on Chinese datasets. While the authors do mention testing on internal code-switching datasets, it would be valuable to see the system's performance on a more diverse range of languages and speech patterns to better understand its broader applicability.

Additionally, the paper does not provide much insight into the computational efficiency or inference latency of the KWS-Whisper system, which could be an important consideration for real-time ASR applications. A more in-depth analysis of the model's resource usage and scalability would help readers better assess its practical viability.

Overall, the research presented in this paper is a significant contribution to the field of ASR, particularly in the context of zero-shot language understanding and bridging the performance gap between human and machine transcription. The authors' innovative use of keyword spotting to enhance a state-of-the-art model like Whisper is a compelling approach that warrants further exploration and validation across a wider range of scenarios.

Conclusion

The KWS-Whisper system introduced in this paper represents a significant advancement in automatic speech recognition (ASR) technology, addressing the critical challenge of accurately recognizing rare named entities and specialized terminology. By leveraging the Whisper model and integrating an open-vocabulary keyword spotting component, the researchers have developed a versatile ASR system that can be easily adapted to user-specific needs and integrated with other error correction methods.

The strong performance of KWS-Whisper on the evaluated datasets, particularly in terms of improved entity recall, demonstrates the potential of this approach to enhance the overall accuracy and robustness of ASR systems. As the use of voice interfaces continues to grow in a wide range of applications, from personal digital assistants to specialized industry tools, the ability to accurately transcribe rare and specialized vocabulary will become increasingly important.

Overall, this research represents an important step forward in bridging the gap between human and machine speech recognition, with promising implications for fields such as zero-shot language understanding and automatic transcription. The authors' innovative use of multitask learning and their modular design approach open up exciting avenues for further development and integration of the KWS-Whisper system in real-world applications.