Language Models and Retrieval Augmented Generation for Automated Structured Data Extraction from Diagnostic Reports

Overview

• The paper aims to develop and evaluate an automated system for extracting structured clinical information from unstructured radiology and pathology reports.
• The system uses open-sourced large language models (LMs) and retrieval-augmented generation (RAG) techniques.
• The study assesses the impact of various model configuration variables on the extraction performance.

Plain English Explanation

The paper describes a system that can automatically extract important clinical information from doctors' notes and reports. These notes are often written in a free-form, unstructured way, making it difficult for computers to understand and extract the key details.

The researchers developed a pipeline that uses powerful language models - large AI systems trained on massive amounts of text data - to analyze the reports and pull out specific pieces of information. For example, they tested the system on extracting brain tumor staging scores from radiology reports and detecting genetic mutations from pathology reports.

They found that the best-performing models could achieve over 98% accuracy in extracting the brain tumor scores and over 90% accuracy for the genetic mutation status. The key factors that improved performance were using larger, more recently developed language models that had been fine-tuned on medical data, as well as carefully crafting the prompts (instructions) given to the models.

This shows that these AI language models have significant potential to automate the extraction of structured medical data from unstructured reports. This could save clinicians time and effort, and make it easier to gather and analyze large datasets for research. However, the researchers note that careful model selection, prompt engineering, and semi-automated optimization are critical for getting the best results.

Technical Explanation

The paper presents an automated pipeline for extracting structured clinical information from unstructured radiology and pathology reports using large language models (LMs) and retrieval-augmented generation (RAG) techniques.

The researchers tested the system on two clinical datasets: 7,294 radiology reports annotated for Brain Tumor Reporting and Data System (BT-RADS) scores, and 2,154 pathology reports annotated for isocitrate dehydrogenase (IDH) mutation status. They systematically evaluated the impact of various model configuration variables, including model size, quantization, prompting strategies, output formatting, and inference parameters.

The best-performing models achieved over 98% accuracy in extracting BT-RADS scores from radiology reports and over 90% accuracy for IDH mutation status extraction from pathology reports. The top-performing model was a medical fine-tuned version of the Llama3 language model. Larger, newer, and domain-specific fine-tuned models consistently outperformed older and smaller models.

Model quantization (a technique to reduce model size and inference time) had minimal impact on performance. Carefully crafted prompting strategies, such as few-shot learning, significantly improved accuracy. The RAG approach improved performance for complex pathology reports but not for the shorter radiology reports.

Critical Analysis

The researchers acknowledge several limitations and areas for further research:

• The study was limited to two specific clinical tasks (BT-RADS extraction and IDH mutation status), and the findings may not generalize to other types of clinical information extraction.
• The performance of the models was evaluated on pre-annotated datasets, which may not reflect real-world challenges such as ambiguous or incomplete reports.
• The study did not explore the impact of combining multiple LMs or ensembling different approaches, which could potentially further improve performance.
• The researchers note that the semi-automated optimization process required for optimal performance may limit the scalability and accessibility of the system.

Additionally, while the results are promising, the researchers emphasize that careful model selection, prompt engineering, and semi-automated optimization are critical for achieving the best performance. This suggests that the deployment of such systems in real-world clinical settings may require significant effort and expertise.

Conclusion

The paper demonstrates the significant potential of open-sourced large language models and retrieval-augmented generation techniques for automated extraction of structured clinical data from unstructured medical reports. The best-performing models achieved over 98% accuracy on the tested tasks, highlighting the feasibility of these approaches for practical use in research workflows.

However, the researchers caution that careful model selection, prompt engineering, and semi-automated optimization are essential for optimal performance. These findings suggest that human-machine collaboration, with clinicians and data scientists working together, may be the most reliable and scalable approach for leveraging these technologies in healthcare data extraction.