Can Authorship Attribution Models Distinguish Speakers in Speech Transcripts?

Overview

• This paper explores the challenge of authorship verification for transcribed speech, which differs from written text analysis.
• Many stylistic features used in written text attribution are not informative for transcribed speech, but it may exhibit other patterns like filler words and backchannels that could be characteristic of different speakers.
• The paper proposes a new benchmark for speaker attribution focused on human-transcribed conversational speech transcripts, controlling for topic to avoid spurious associations.
• The authors establish the state of the art on this benchmark by comparing neural and non-neural baselines, finding that written text attribution models perform worse as conversational topic is increasingly controlled.

Plain English Explanation

Authorship verification is the task of determining if two different writing samples were written by the same person. This paper explores a new challenge in this area - verifying the authorship of transcribed speech rather than written text.

When analyzing written text, things like punctuation and capitalization can provide clues about the author. But these stylistic features aren't as informative for transcribed speech. However, transcribed speech does have other patterns, like use of filler words and backchannels (e.g. "um," "uh-huh"), that may be characteristic of different speakers.

To study this, the researchers created a new dataset of human-transcribed conversational speech. They carefully designed the dataset to control for the topic of the conversations, so that any speaker differences detected weren't just due to talking about different subjects.

The researchers then tested a variety of neural and non-neural models for attributing the transcribed speech to different speakers. They found that models trained on written text performed surprisingly well in some cases, but their performance dropped significantly as the conversational topic was more controlled.

The paper also looks at how the style of the transcription itself impacts the model performance, and whether fine-tuning on speech transcripts can improve the results.

Technical Explanation

The paper proposes a new benchmark for speaker attribution focused on human-transcribed conversational speech. To construct this benchmark, the authors used conversation prompts and ensured that speakers participated in the same conversations, in order to control for topic and limit spurious associations between speakers and content.

The authors compare a suite of neural and non-neural baselines on this new benchmark, including models trained on written text as well as models fine-tuned on the speech transcripts. They find that while written text attribution models achieve surprisingly good performance in certain settings, their performance drops markedly as conversational topic is increasingly controlled.

The paper also presents analyses of the impact of transcription style on model performance, as well as the ability of fine-tuning on speech transcripts to improve performance. Overall, the results highlight the novel challenges posed by the attribution of transcribed speech compared to written text.

Critical Analysis

The paper acknowledges some limitations of the proposed benchmark, including the reliance on human-transcribed speech rather than automatically transcribed speech, which may differ in stylistic patterns. Additionally, the authors note that the benchmark only covers a limited set of conversation prompts and speakers, and that further work is needed to test the generalization of the models.

One potential issue not addressed in the paper is the potential bias introduced by the human transcribers, whose own stylistic preferences and idiosyncrasies could be reflected in the transcripts. This could complicate the task of distinguishing between different speakers' speech patterns.

Additionally, the paper does not explore the potential impact of speaker demographics, such as age, gender, or regional accent, on the performance of the attribution models. These factors could also play a role in shaping the linguistic patterns observed in the transcripts.

Overall, the paper presents a valuable contribution to the field of authorship verification, highlighting the unique challenges and considerations involved in analyzing transcribed speech. The new benchmark and the insights gained from the comparative analysis of baselines provide a solid foundation for future research in this area.

Conclusion

This paper tackles the novel challenge of authorship verification for transcribed speech, which differs from the more typical task of written text attribution. By creating a carefully controlled benchmark dataset and evaluating a range of models, the authors demonstrate that many stylistic features used in written text attribution are not informative for transcribed speech, but that other patterns, such as filler words and backchannels, may be characteristic of different speakers.

The findings from this research could have important implications for applications like speaker diarization, where accurately attributing speech to individual speakers is crucial, as well as generating citations and question-answering systems that rely on accurate speaker attribution. The insights from this work can help guide future research and development in these areas.