The emergence of Large Language Models (LLM) as a tool in literature reviews: an LLM automated systematic review

Overview

• This study aims to summarize the use of Large Language Models (LLMs) in the process of creating a scientific review.
• It looks at the different stages of a review that can be automated and assesses the current state-of-the-art research projects in this field.

Plain English Explanation

The paper explores how Large Language Models (LLMs) can be used to help streamline the scientific review process. Scientific reviews are an important part of advancing knowledge, but they can be time-consuming and labor-intensive to create.

The researchers wanted to see which parts of the review process could be automated using LLMs, which are powerful AI models trained on vast amounts of text data. They looked at things like searching for relevant publications, extracting key data, and even generating draft text for the review itself.

The results suggest that LLMs can indeed help streamline many aspects of the review process, with the potential to make reviews more efficient and less resource-intensive to produce. However, the technology is still relatively new, and there is more research needed to fully assess the efficacy of LLMs for systematic reviews.

Technical Explanation

The researchers conducted a systematic search in June 2024 across several academic databases (PubMed, Scopus, Dimensions, Google Scholar) to identify research projects related to using LLMs for scientific review automation.

Screening and data extraction were assisted by an LLM add-on to the Covidence software, which leveraged the GPT-4 model. The extracted data was then cleaned and processed using ChatGPT, and ChatGPT and Scite.ai were used to draft most components of the manuscript (excluding methods and discussion).

Out of 3,788 articles retrieved, 172 studies were deemed eligible for the final review. The results showed that ChatGPT and other GPT-based LLMs were the most dominant architectures used for review automation (73.2% of the studies). Many of the projects focused on automating specific stages of the review process, such as searching for relevant publications (34.9%) and extracting key data (31.4%).

When comparing the performance of GPT-based and BERT-based models, the GPT-based models were generally more accurate for data extraction tasks (mean precision of 83.0% and recall of 86.0%), but slightly less accurate for title and abstract screening (mean accuracy of 77.3%).

Critical Analysis

The study provides a comprehensive overview of the current state of research on using LLMs for scientific review automation. The authors acknowledge that while the results are promising, the technology is still relatively new and there is more work needed to fully assess its efficacy for systematic reviews.

One potential limitation is that the search was conducted in June 2024, so the findings may not fully reflect the latest developments in this rapidly evolving field. Additionally, the study focused on a broad range of review automation projects, rather than deeper dives into the performance and limitations of specific LLM-based approaches.

Further research could explore the impact of different LLM architectures, as well as how LLMs perform compared to human reviewers across the various stages of the review process. Investigating the potential biases or errors introduced by LLMs in the review process would also be an important area for future study.

Conclusion

This study provides a valuable snapshot of the current state of research on using Large Language Models (LLMs) to automate various stages of the scientific review process. The results suggest that LLMs have significant potential to streamline and improve the efficiency of reviews, but there is still more work needed to fully assess their efficacy and limitations.

As the field of LLM research continues to rapidly evolve, it will be important for researchers and practitioners to closely follow the latest developments and explore how these powerful AI models can be leveraged to enhance the scientific review process and ultimately advance knowledge in a more efficient and effective manner.