MedPromptX: Grounded Multimodal Prompting for Chest X-ray Diagnosis

Introduction

The provided text discusses how emerging machine learning and deep learning advancements are assisting radiologists in detecting chest X-ray abnormalities. While traditional diagnosis based solely on imaging data can be effective, incorporating patients' clinical history can significantly improve diagnostic outcomes, highlighting the importance of multimodal approaches.

The integration of electronic health records (EHR) has been challenged by its inherent incompleteness, with missing values and lack of normal ranges for laboratory tests complicating the interpretation of medical datasets. Large language models (LLMs) have shown promise in clinical prediction by fine-tuning with prompts leveraging structured EHR data.

The emergence of generalist models like BiomedGPT represents a major advancement in biomedical AI, handling various tasks across modalities and surpassing state-of-the-art results. Visual grounding techniques have also demonstrated progress in medical imaging, particularly in automating associations between image features and descriptive reports in CT imaging.

Despite these advancements, there remains a gap in the integration of multimodal data for enhancing diagnostic accuracy in chest X-ray analysis. The few-shot prompting technique enables rapid adaptation to new diagnostic tasks with minimal labeled data, empowering medical professionals to efficiently use accurate diagnostic solutions tailored to specific patient cases. However, the quality and quantity of the few-shot data play a pivotal role in influencing performance.

The text introduces MedPromptX, a novel diagnostic model for chest X-ray images that integrates multimodal LLMs, few-shot prompting, and visual grounding to address the challenge of incomplete EHR data and enhance diagnostic accuracy. Additionally, the authors propose a dynamic proximity selection (DPS) technique that refines the few-shot data in real-time, allowing the model to capture the nuanced relationships between patient history and patient outcomes.

Methodology

This paper presents MedPromptX, a framework for medical diagnosis using a four-phase process. The key components are:

Visual Grounding (VG): This module uses Grounding DINO to detect regions of interest in chest X-ray images and generate grounded image representations.

Dynamic Proximity Selection (DPS): This method refines the candidate samples by ordering them based on their proximity to the query sample, using a distance function like cosine similarity. This enhances the robustness and adaptability of the few-shot learning approach.

Multimodal Large Language Model (MLLM): The paper uses the Med-Flamingo model, based on the Flamingo framework, to process the interleaved image and text inputs and predict the likelihood of the text sequence conditioned on the accompanying image.

The authors also describe the creation of the MedPromptX-VQA dataset, which combines data from the MIMIC-IV and MIMIC-CXR databases to focus on patients with chest X-ray procedures. The dataset covers 12 pathological conditions and is structured to support in-context learning tasks.

Experimental Setup

The paper employed a randomized order strategy for the input sequences across several state-of-the-art models, including Med-Flamingo, OpenFlamingo, BioMedLM, and Clinical-T5-Large. The number of few-shot samples was consistent at 6 across all models, chosen randomly. For MedPromptX, the number of few-shot candidates is dynamically reduced by the DPS technique, resulting in a unique configuration for each query instance. The experiments were conducted using NVIDIA A100-SXM GPUs with 40GB of dedicated memory.

The paper describes the models used in the study. Med-Flamingo and OpenFlamingo ingest interleaved image and text, excluding electronic health record (EHR) data, while BioMedLM and Clinical-T5 use text, including EHR data. MedPromptX is the only model that processes interleaved grounded or original image and EHR text prompt.

The paper provides examples of the different prompt types used in the study: image and text, EHR text, and image with EHR text.

Results and Discussion

The results in this paper show the complexity of medical diagnosis, where combining multiple data sources like imaging and clinical text can lead to better model performance. The MedPromptX approach integrates these diverse data sources, but initial attempts faced challenges. The use of Data Prompt Synthesis (DPS) and Visual Guidance (VG) strategies helped overcome these obstacles.

DPS reduces the number of ambiguous examples, allowing the model to learn more effectively from limited data. In contrast, a random configuration of Feature Prompts (FP) can introduce biases. VG enables the model to focus on relevant regions within images by generating semantic output embeddings, rather than working with raw pixel data. However, the performance gap when using VG alone may be due to the model being trained on general domain data instead of chest X-ray images.

The paper's ablation studies show that initializing DPS with more examples improves generalization, though a 6-shot setting balances performance and class inclusion. Zero-shot assessment was not feasible due to model hallucination, underscoring the need for FP. Adjusting the DPS threshold can significantly impact performance, and using multimodal similarity enhances the instance selection process compared to single-modality approaches.

Conclusion

MedPromptX is a new model that combines a patient's clinical history with imaging data to accurately diagnose chest X-rays. The model addresses issues with incomplete medical data, adapting to new patients with limited labeled data, and detecting abnormalities in X-ray images. While the paper did not explore using fine-tuned backbones, which could further improve performance, future work may focus on accessing diverse, well-annotated datasets. Additionally, rigorous clinical trials and real-world deployment are necessary to validate the model's effectiveness and clinical usefulness.

Appendix 0.A Appendix