A Multi-Center Study on the Adaptability of a Shared Foundation Model for Electronic Health Records

Overview

• Foundation models hold promise for transforming healthcare AI by providing modular components that can be easily adapted to different healthcare tasks, making AI development more scalable and cost-effective.
• Structured EHR (electronic health record) foundation models, trained on medical records from millions of patients, have shown benefits like increased performance with fewer training labels and improved robustness to distribution shifts.
• However, questions remain about the feasibility of sharing these models across different hospitals and their performance for local task adaptation.

Plain English Explanation

Foundation models are powerful AI systems that can be used as building blocks for a wide range of applications. In the healthcare domain, foundation models for structural health monitoring and global contrastive training of multimodal EHR data have shown promise.

This study explored the use of a specific structured EHR foundation model, $FM_{SM}$, which was trained on medical records from 2.57 million patients at Stanford Medicine. The researchers wanted to see how well this pre-trained model could be adapted to healthcare tasks at other hospitals, without having to start from scratch each time.

They tested the model's adaptability on EHR data from The Hospital for Sick Children and the MIMIC-IV dataset. The results were promising - the adapted $FM_{SM}$ model was able to match the performance of models trained locally on all the data, while needing much less training data. This suggests that foundation models can be a cost-effective way to develop healthcare AI applications, by providing a strong starting point that can be fine-tuned for specific tasks and locations.

Technical Explanation

The study examined the adaptability of a recently released structured EHR foundation model, $FM_{SM}$, which was trained on longitudinal medical record data from 2.57 million Stanford Medicine patients.

Experiments were conducted using EHR data from The Hospital for Sick Children and the MIMIC-IV dataset. The researchers assessed two aspects of adaptability:

1. Adaptability via continued pretraining on local data
2. Task adaptability compared to baselines of training models from scratch at each site, including a locally trained foundation model

The performance of the adapted models was evaluated on 8 clinical prediction tasks. In both datasets, adapting the off-the-shelf $FM_{SM}$ model matched the performance of locally trained Gradient Boosting Machine (GBM) models, while providing a 13% improvement in settings with few task-specific training labels.

With continued pretraining on local data, label efficiency substantially improved, such that $FM_{SM}$ required less than 1% of the training examples to match the fully trained GBM's performance. This continued pretraining was also 60 to 90% more sample-efficient than training local foundation models from scratch.

Critical Analysis

The study provides promising evidence for the adaptability of structured EHR foundation models across different healthcare institutions. By showing that the pre-trained $FM_{SM}$ model can be effectively adapted to local tasks with minimal additional training, the research suggests that foundation models could help streamline the development of healthcare AI applications and make them more scalable and cost-effective.

However, the study also highlights the need for further research on the feasibility of sharing these models across hospitals. While the adapted $FM_{SM}$ model performed well, questions remain about how it would perform compared to locally trained models in more diverse healthcare settings, and whether there are any barriers to sharing the model due to data privacy or other concerns.

Additionally, the study only examined 8 clinical prediction tasks, and it would be valuable to explore the model's adaptability to a wider range of healthcare applications, such as predicting major postoperative complications or effective learning benchmarks for medical data.

Conclusion

This study demonstrates the potential of adapting structured EHR foundation models to improve the development of healthcare AI applications. By leveraging a pre-trained model like $FM_{SM}$, researchers and developers can save time and resources while achieving strong performance on a variety of clinical prediction tasks.

The findings suggest that foundation models could play a key role in advancing healthcare AI, by providing modular components that can be easily adapted to different use cases. As the field continues to explore the capabilities and limitations of these powerful AI systems, this research highlights the promise of foundation models for transforming healthcare through more scalable and cost-effective AI development.