DiaPer: End-to-End Neural Diarization with Perceiver-Based Attractors

Overview

• Until recently, speaker diarization systems were built using a cascaded approach, which had limitations in handling overlapped speech and had complex pipelines.
• End-to-end models have gained popularity as they can address these limitations.
• One successful end-to-end model is EEND-EDA, which uses an encoder-decoder-based attractor network.
• This paper proposes replacing the attractor network in EEND-EDA with a Perceiver-based module, resulting in a new model called DiaPer.

Plain English Explanation

The paper describes a new approach to speaker diarization, which is the task of identifying who is speaking when in an audio recording. Traditionally, speaker diarization systems were built using a multi-step process, where different components were chained together. This made the systems complex and limited their ability to handle situations where multiple people were speaking at the same time.

More recently, end-to-end models have been developed that can perform the entire diarization task in a single step. One successful end-to-end model is called EEND-EDA, which uses an encoder-decoder-based attractor network to identify the speakers.

In this paper, the researchers propose a new model called DiaPer that replaces the attractor network in EEND-EDA with a Perceiver-based module. The Perceiver model is a type of deep learning architecture that can efficiently process complex, high-dimensional data. The researchers show that this modification leads to better performance on the widely-used Callhome dataset, more accurate speaker count estimation, and faster inference times compared to the original EEND-EDA model.

Technical Explanation

The paper introduces a new end-to-end speaker diarization model called DiaPer, which builds upon the EEND-EDA architecture. The key innovation is the replacement of the encoder-decoder-based attractor network in EEND-EDA with a Perceiver-based module.

The Perceiver model is a type of attention-based neural network that can efficiently process high-dimensional, complex data. By incorporating the Perceiver module, the researchers show that DiaPer achieves better performance on the Callhome dataset compared to EEND-EDA, with improved accuracy in estimating the number of speakers in a conversation. Additionally, DiaPer demonstrates faster inference times.

The paper also provides a comprehensive evaluation of DiaPer against other state-of-the-art speaker diarization methods across more than ten public wide-band datasets. The results demonstrate that DiaPer reaches remarkable performance while maintaining a very lightweight design.

Critical Analysis

The paper presents a well-designed study that introduces a novel speaker diarization model with clear advantages over the previous state-of-the-art EEND-EDA approach. The researchers have thoroughly evaluated their model's performance across multiple datasets and compared it to various other methods.

One potential limitation of the study is the lack of detailed analysis on the specific architectural choices and their impact on the model's performance. While the Perceiver-based module is a key contribution, the paper could have provided more insight into why this particular architecture was selected and how it compares to other attention-based or end-to-end models, such as DiarizationLM.

Additionally, the paper does not delve into potential real-world deployment challenges or limitations of the DiaPer model, such as its performance in noisy environments, its ability to handle speaker overlaps, or its scalability to large-scale datasets. Addressing these aspects in future research could further strengthen the practical relevance of the proposed approach.

Conclusion

The paper introduces DiaPer, a novel end-to-end speaker diarization model that outperforms the previous state-of-the-art EEND-EDA approach. By replacing the attractor network with a Perceiver-based module, DiaPer achieves better performance on the Callhome dataset, more accurate speaker count estimation, and faster inference times.

The comprehensive evaluation of DiaPer across multiple public datasets demonstrates its remarkable performance and lightweight design, making it a promising candidate for real-world speaker diarization applications. The open-sourcing of the DiaPer code and pre-trained models further contributes to the accessibility and reproducibility of the research, which can benefit the broader speech processing community.