Evaluating Large Language Models for Public Health Classification and Extraction Tasks

Overview

• This paper explores the potential of large language models (LLMs) to support public health tasks, such as classifying and extracting information from free text.
• The researchers evaluated the performance of several open-weight LLMs (7-70 billion parameters) on a range of public health-related tasks, including assessing health burden, epidemiological risk factors, and public health interventions.
• The study found that the LLama-3-70B-Instruct model performed the best overall, outperforming other models on 15 out of 17 tasks.
• The researchers also compared the performance of LLMs to GPT-4, finding that they achieved comparable results on a subset of 12 tasks.

Plain English Explanation

Large language models (LLMs) are powerful artificial intelligence systems that can be used to process and understand natural language. Researchers have been exploring how these models can be applied to support public health efforts, such as analyzing text-based information related to health issues, disease outbreaks, and public health interventions.

In this study, the researchers evaluated the performance of several LLMs on a variety of public health-related tasks. They used a combination of existing datasets and newly annotated datasets to test the models' ability to classify and extract information from free-form text. For example, they asked the models to identify whether a piece of text was discussing the health burden of a particular disease, the risk factors associated with an illness, or the details of a public health intervention.

The researchers found that the LLama-3-70B-Instruct model performed the best overall, outperforming other models on the majority of the tasks. However, they also noted that the models struggled with some more challenging tasks, such as correctly classifying whether a piece of text was discussing contact tracing efforts.

When the researchers compared the LLMs to the powerful GPT-4 model, they found that the LLMs achieved comparable results on a subset of the tasks. This suggests that these large language models could potentially be useful tools for public health experts to help them extract information from a wide variety of text-based sources, such as news reports, research papers, and social media posts.

Overall, this research provides promising evidence that LLMs may be able to support public health surveillance, research, and interventions by automating the analysis of large amounts of textual data. However, the researchers also acknowledge that there is still room for improvement, particularly on more complex tasks.

Technical Explanation

The researchers in this study evaluated the performance of several open-weight large language models (LLMs) on a range of public health-related tasks. They combined six externally annotated datasets and seven new internally annotated datasets to create a comprehensive evaluation framework covering three main areas: health burden, epidemiological risk factors, and public health interventions.

The researchers initially tested five open-weight LLMs, ranging from 7 to 70 billion parameters, using a zero-shot in-context learning approach. They found that the LLama-3-70B-Instruct model outperformed the other LLMs, achieving the best results on 15 out of the 17 tasks based on micro-F1 scores.

The performance of the LLMs varied significantly across the different tasks. While all the models scored above 80% micro-F1 on some tasks, such as classifying gastrointestinal illness, they all scored below 60% micro-F1 on more challenging tasks, such as contact classification.

For a subset of 12 tasks, the researchers also evaluated the performance of the GPT-4 model, which is known for its strong language understanding capabilities. They found that the LLama-3-70B-Instruct model performed equally well or better than GPT-4 on 6 of the 12 tasks.

Overall, the results of this study suggest that LLMs may be valuable tools for public health experts to extract information from a wide variety of free-text sources, supporting efforts in areas such as public health surveillance, research, and interventions. However, the researchers also acknowledge that there is still room for improvement, particularly on more complex tasks.

Critical Analysis

The researchers in this study provide a comprehensive evaluation of LLMs for public health-related tasks, which is an important contribution to the growing body of research on the use of large language models in healthcare and medicine.

One strength of the study is the diverse range of tasks and datasets used to assess the LLMs' performance. By combining existing and newly annotated datasets, the researchers were able to create a robust evaluation framework that covered a variety of public health-related topics and challenges.

However, the study also highlights the limitations of current LLMs, particularly on more complex tasks. The researchers noted that all the models struggled with certain tasks, such as contact classification, which may require more specialized domain knowledge or reasoning capabilities.

Additionally, the researchers did not provide detailed explanations for the performance differences between the LLMs and GPT-4. It would be interesting to know more about the specific strengths and weaknesses of each model, as well as the potential reasons behind their relative performance on the different tasks.

Another area for further research could be exploring ways to fine-tune or adapt the LLMs to improve their performance on public health-related tasks. The survey of large language models in medicine highlights the potential benefits of specialized training or fine-tuning for specific medical applications.

Overall, this study provides a valuable contribution to the ongoing research on the use of large language models in healthcare and public health. While the results are promising, they also underscore the need for continued development and refinement of these models to fully realize their potential in supporting critical public health efforts.

Conclusion

This study presents a comprehensive evaluation of the performance of large language models (LLMs) on a range of public health-related tasks, including classifying and extracting information from free-form text. The researchers found that the LLama-3-70B-Instruct model outperformed other open-weight LLMs, as well as the powerful GPT-4 model, on many of the tasks.

The results suggest that LLMs may be valuable tools for public health experts to help them process and analyze large amounts of text-based information, supporting efforts in areas such as disease surveillance, risk factor identification, and intervention planning. However, the researchers also acknowledge the limitations of current LLMs, particularly on more complex tasks, and emphasize the need for continued development and refinement of these models to fully realize their potential in the public health domain.

Overall, this study provides an important contribution to the ongoing research on the use of large language models in healthcare and medicine, and it highlights the promising opportunities as well as the challenges that lie ahead in leveraging these powerful AI systems to support critical public health initiatives.