OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer

Read original: arXiv:2401.16658 - Published 8/28/2024 by Yifan Peng, Jinchuan Tian, William Chen, Siddhant Arora, Brian Yan, Yui Sudo, Muhammad Shakeel, Kwanghee Choi, Jiatong Shi, Xuankai Chang and 2 others

🗣️

Overview

Introduces a new version of the Open Whisper-Style Speech Model (OWSM v3.1)
Focuses on improving the performance and efficiency of the model compared to previous versions
Uses an innovative "E-Branchformer" architecture to achieve better and faster speech recognition

Plain English Explanation

The paper presents a new and improved version of the Open Whisper-Style Speech Model (OWSM), called OWSM v3.1. The main goal of this research is to develop a more accurate and efficient speech recognition model that can be used in a variety of applications, such as voice assistants, transcription services, and more.

The key innovations in OWSM v3.1 are the use of an "E-Branchformer" architecture, which is a type of neural network designed to process speech data more effectively. This new architecture allows the model to capture important features from the input speech data more accurately and efficiently, leading to better overall performance.

Furthermore, the researchers have made various other improvements to the model, such as optimizing the training process and incorporating techniques from related fields like natural language processing. These enhancements have resulted in OWSM v3.1 being faster and more accurate than previous versions of the model, making it a more practical and useful tool for real-world applications.

Technical Explanation

The paper introduces OWSM v3.1, which builds upon the previous versions of the Open Whisper-Style Speech Model (OWSM v1) and (OWSM v2). The key innovation in OWSM v3.1 is the use of an "E-Branchformer" architecture, which is a novel neural network design that improves the model's ability to process and understand speech data.

The E-Branchformer architecture consists of multiple branches, each of which focuses on extracting different types of features from the input speech data. This allows the model to capture a more comprehensive representation of the speech signal, leading to better overall performance in speech recognition tasks.

In addition to the architectural changes, the researchers have also explored various other techniques to enhance the model's capabilities, such as:

These combined efforts have resulted in OWSM v3.1 achieving state-of-the-art performance on a range of speech recognition benchmarks, while also being more efficient and faster than previous versions of the model.

Critical Analysis

The paper presents a thorough and well-designed study, with a clear focus on improving the performance and efficiency of the OWSM model. The researchers have addressed several known limitations of previous versions, such as the need for more robust feature extraction and the desire for faster inference times.

However, the paper does not explicitly discuss the potential limitations or caveats of the E-Branchformer architecture. It would be helpful to understand the specific trade-offs or design choices that were made, and how they might impact the model's performance in certain scenarios or with certain types of speech data.

Additionally, the paper could have explored the potential impact of incorporating Whisper-based models to further enhance the model's robustness and performance. This could be an interesting area for future research.

Overall, the paper presents a compelling and well-executed study that advances the state of the art in open-source speech recognition models. The researchers have demonstrated the potential of the E-Branchformer architecture and have provided a strong foundation for future work in this area.

Conclusion

The OWSM v3.1 paper introduces a significant improvement to the Open Whisper-Style Speech Model, with a focus on enhancing performance and efficiency. The key innovation is the use of an "E-Branchformer" architecture, which allows the model to capture more comprehensive representations of speech data, leading to better overall recognition accuracy.

The researchers have also explored various other techniques to further improve the model, such as incorporating insights from related fields and optimizing the training process. These combined efforts have resulted in OWSM v3.1 achieving state-of-the-art performance on a range of speech recognition benchmarks, while also being more efficient and faster than previous versions of the model.

The findings of this study have important implications for the development of practical and widely-applicable speech recognition systems, which are essential for a variety of real-world applications, from voice assistants to transcription services. By continuing to push the boundaries of open-source speech recognition, this research helps to make these powerful technologies more accessible and impactful for the broader community.

This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!

Follow @aimodelsfyi on 𝕏 →

Related Papers

🗣️

OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer

Yifan Peng, Jinchuan Tian, William Chen, Siddhant Arora, Brian Yan, Yui Sudo, Muhammad Shakeel, Kwanghee Choi, Jiatong Shi, Xuankai Chang, Jee-weon Jung, Shinji Watanabe

Recent studies have highlighted the importance of fully open foundation models. The Open Whisper-style Speech Model (OWSM) is an initial step towards reproducing OpenAI Whisper using public data and open-source toolkits. However, previous versions of OWSM (v1 to v3) are still based on standard Transformer, which might lead to inferior performance compared to state-of-the-art speech encoder architectures. This work aims to improve the performance and efficiency of OWSM without additional data. We present a series of E-Branchformer-based models named OWSM v3.1, ranging from 100M to 1B parameters. OWSM v3.1 outperforms its predecessor, OWSM v3, in most evaluation benchmarks, while showing an improved inference speed of up to 25%. We further reveal the emergent ability of OWSM v3.1 in zero-shot contextual biasing speech recognition. We also provide a model trained on a subset of data with low license restrictions. We will publicly release the code, pre-trained models, and training logs.

8/28/2024

On the Effects of Heterogeneous Data Sources on Speech-to-Text Foundation Models

Jinchuan Tian, Yifan Peng, William Chen, Kwanghee Choi, Karen Livescu, Shinji Watanabe

The Open Whisper-style Speech Model (OWSM) series was introduced to achieve full transparency in building advanced speech-to-text (S2T) foundation models. To this end, OWSM models are trained on 25 public speech datasets, which are heterogeneous in multiple ways. In this study, we advance the OWSM series by introducing OWSM v3.2, which improves on prior models by investigating and addressing the impacts of this data heterogeneity. Our study begins with a detailed analysis of each dataset, from which we derive two key strategies: data filtering with proxy task to enhance data quality, and the incorporation of punctuation and true-casing using an open large language model (LLM). With all other configurations staying the same, OWSM v3.2 improves performance over the OWSM v3.1 baseline while using 15% less training data.

6/14/2024

OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification

Yifan Peng, Yui Sudo, Muhammad Shakeel, Shinji Watanabe

There has been an increasing interest in large speech models that can perform multiple tasks in a single model. Such models usually adopt an encoder-decoder or decoder-only architecture due to their popularity and good performance in many domains. However, autoregressive models can be slower during inference compared to non-autoregressive models and also have potential risks of hallucination. Though prior studies observed promising results of non-autoregressive models for certain tasks at small scales, it remains unclear if they can be scaled to speech-to-text generation in diverse languages and tasks. Inspired by the Open Whisper-style Speech Model (OWSM) project, we propose OWSM-CTC, a novel encoder-only speech foundation model based on Connectionist Temporal Classification (CTC). It is trained on 180k hours of public audio data for multilingual automatic speech recognition (ASR), speech translation (ST), and language identification (LID). Compared to encoder-decoder OWSM, our OWSM-CTC achieves competitive results on ASR and up to 24% relative improvement on ST, while it is more robust and 3 to 4 times faster for inference. OWSM-CTC also improves the long-form ASR result with 20x speed-up. We will publicly release our code, pre-trained model, and training logs to promote open science in speech foundation models.

8/28/2024

🏋️

A Multitask Training Approach to Enhance Whisper with Contextual Biasing and Open-Vocabulary Keyword Spotting

Yuang Li, Min Zhang, Chang Su, Yinglu Li, Xiaosong Qiao, Mengxin Ren, Miaomiao Ma, Daimeng Wei, Shimin Tao, Hao Yang

The recognition of rare named entities, such as personal names and terminologies, is challenging for automatic speech recognition (ASR) systems, especially when they are not frequently observed in the training data. In this paper, we introduce keyword spotting enhanced Whisper (KWS-Whisper), a novel ASR system that leverages the Whisper model and performs open-vocabulary keyword spotting (OV-KWS) on the hidden states of the Whisper encoder to recognize user-defined named entities. These entities serve as prompts for the Whisper decoder. To optimize the model, we propose a multitask training approach that learns OV-KWS and contextual-ASR tasks. We evaluate our approach on Chinese Aishell hot word subsets and two internal code-switching test sets and show that it significantly improves the entity recall compared to the original Whisper model. Moreover, we demonstrate that the OV-KWS can be a plug-and-play module to enhance the ASR error correction methods and frozen Whisper models.

6/7/2024