Coding historical causes of death data with Large Language Models

Overview

• This paper investigates using pre-trained large language models (LLMs) to automatically assign ICD-10 codes to historical causes of death.
• ICD-10 codes are a standardized system for classifying diseases and causes of death, traditionally assigned manually by coding experts.
• The authors evaluate the performance of GPT-3.5, GPT-4, and Llama 2 LLMs on the HiCaD dataset, which contains over 19,000 historical causes of death.
• The findings show that LLMs can achieve reasonable accuracy, but standard machine learning techniques still outperform them for this task.

Plain English Explanation

The paper explores using advanced AI language models to automatically categorize historical causes of death. Causes of death are often recorded in complex, narrative form, making them difficult to classify.

The researchers tested three powerful language models - GPT-3.5, GPT-4, and Llama 2 - on a dataset of over 19,000 historical causes of death. These models are trained on vast amounts of text data and can generate human-like language. The goal was to see if they could accurately assign standardized ICD-10 medical codes to the causes of death.

The results were mixed. The language models were able to correctly code between 40-83% of the causes, depending on the model. However, traditional machine learning techniques achieved even higher accuracy at 89%.

The language models performed better on causes that used common medical terms, compared to older, more obscure language. They also did better on short, simple causes versus longer, more complex ones.

Overall, while the language models showed promise, they still struggle to match human experts for this specialized task of historical cause of death coding. The researchers suggest further training or different approaches may be needed to truly automate this process.

Technical Explanation

The paper evaluates the feasibility of using large language models (LLMs) to automate the assignment of ICD-10 codes to historical causes of death. ICD-10 is a standardized system for classifying diseases and causes of mortality, traditionally assigned manually by coding experts.

The authors tested three prominent LLMs - GPT-3.5, GPT-4, and Llama 2 - on the HiCaD dataset, which contains over 19,000 historical causes of death recorded in civil death registers in the UK between 1861-1901. These causes often contain complex, narrative language that makes manual coding challenging.

The results showed that the LLMs could achieve reasonably high accuracy, correctly coding 69%, 83%, and 40% of causes for GPT-3.5, GPT-4, and Llama 2 respectively. However, the authors found that standard machine learning techniques could achieve even higher accuracy at 89%.

Further analysis revealed that the LLMs performed better on causes containing terms still in common medical use, compared to more archaic language. They also did better on short, simple causes versus longer, more complex ones.

Overall, the findings indicate that while LLMs show promise, they do not currently perform well enough to fully automate the historical ICD-10 coding task. The authors suggest further fine-tuning or alternative frameworks may be needed to reach adequate performance levels.

Critical Analysis

The paper provides a thorough evaluation of using LLMs to assist with medical coding tasks, which is an important step towards automating laborious manual processes. However, the authors acknowledge several limitations and areas for further research.

One key limitation is the narrow scope of the dataset, which only includes historical causes of death from three UK regions. Expanding the evaluation to a broader, more diverse set of medical narratives would be helpful to assess the generalizability of the findings.

Additionally, the authors do not explore the potential benefits of combining LLMs with other techniques, such as structured knowledge bases or ensembles of machine learning models. Such hybrid approaches may be able to achieve higher accuracy than any single method alone.

It would also be worthwhile to investigate the interpretability and explainability of the LLM coding decisions. Understanding the reasoning behind the model's outputs could aid in building trust and acceptance for this type of AI-assisted medical coding.

Overall, this paper represents an important step in advancing our understanding of LLM capabilities for specialized medical tasks. However, further research is needed to fully realize the potential of these models in real-world healthcare applications.

Conclusion

This paper explores the use of pre-trained large language models (LLMs) to automate the assignment of ICD-10 codes to historical causes of death. While the tested LLMs (GPT-3.5, GPT-4, and Llama 2) demonstrated reasonable performance, standard machine learning techniques still outperformed them for this task.

The findings suggest that LLMs struggle with the complex, narrative nature of historical causes of death, particularly when they contain archaic medical terminology. However, the models performed better on short causes using common medical terms, indicating they may have potential for certain coding applications.

Overall, this research highlights the need for further advancements in LLM capabilities, potentially through fine-tuning or hybrid approaches, to reliably automate the complex task of historical medical coding. As AI language models continue to evolve, they may become increasingly valuable tools for streamlining medical documentation and improving healthcare efficiency.