Generative AI for automatic topic labelling

Overview

• Topic modeling has become a prominent tool for studying scientific fields, as it allows for large-scale interpretation of research trends.
• However, the output of topic models is a list of keywords, which requires manual interpretation and labeling.
• This paper aims to assess the reliability of three large language models (LLMs) - flan, GPT-4o, and GPT-4 mini - for topic labeling.

Plain English Explanation

Topic modeling is a technique used to automatically identify the main themes or topics in a large collection of text, such as scientific articles. This can be useful for understanding the trends and patterns in a field of research.

The output of topic models is a list of keywords that represent each topic. However, these keywords can be difficult to interpret and label in a meaningful way. This paper investigated whether large language models - powerful AI systems trained on vast amounts of text data - could be used to automatically label the topics generated by a topic model.

The researchers used a dataset of scientific articles authored by biology professors in Switzerland between 2008 and 2020. They fed this data into a topic modeling algorithm to identify the main research topics. They then tested three different LLMs (flan, GPT-4o, and GPT-4 mini) to see how well they could label these topics based on the keyword lists.

Technical Explanation

The researchers used a BERTopic topic modeling approach to identify the main research topics from a dataset of 34,797 scientific articles authored by 465 biology professors in Switzerland between 2008 and 2020. This generated a list of topic keywords for each research theme.

They then assessed the performance of three large language models - flan, GPT-4o, and GPT-4 mini - in labeling these topics. The LLMs were given the lists of keywords and asked to provide concise 3-word labels that capture the essence of each research topic.

The researchers evaluated the LLM-generated labels both quantitatively and qualitatively. They found that both the GPT models were able to accurately and precisely label the topics based on the keyword lists. The 3-word labels were found to be preferable for capturing the complexity of the research topics.

Critical Analysis

The paper provides a thorough and well-designed evaluation of using LLMs for topic labeling, a common challenge in topic modeling research. The researchers leveraged a large, real-world dataset of scientific articles and employed rigorous quantitative and qualitative methods to assess the LLM performance.

One potential limitation is that the study focused only on the field of biology, so the findings may not generalize to other scientific domains. Additionally, the paper does not address potential biases or limitations of the LLMs themselves, which could impact the reliability of the topic labels.

Further research could explore the use of these LLMs for topic labeling in other fields, as well as investigate ways to integrate the LLM capabilities with the topic modeling process to further improve the overall topic interpretation and understanding.

Conclusion

This study demonstrates the potential of using large language models to reliably and accurately label the topics generated by topic modeling algorithms. By automating the labeling process, the researchers have shown how LLMs can enhance the interpretability and usefulness of topic modeling for studying scientific research trends. These findings have implications for leveraging LLMs to streamline and scale the analysis of large-scale textual data in various research domains.