WsiCaption: Multiple Instance Generation of Pathology Reports for Gigapixel Whole-Slide Images

Overview

• Whole slide images (WSIs) are the foundation of digital pathology for diagnosing and treating carcinomas.
• Pathology report writing is time-consuming and error-prone for inexperienced pathologists.
• This research aims to generate pathology reports automatically from WSIs to reduce workload and improve clinical automation.
• The study curated the largest WSI-text dataset, called PathText, and proposed a multiple instance generative model (MI-Gen) to produce pathology reports from gigapixel WSIs.

Plain English Explanation

Whole slide images (WSIs) are digital scans of tissue samples that pathologists use to diagnose and treat cancer. Writing detailed pathology reports based on these WSIs is a laborious and error-prone process, especially for less experienced pathologists. To address this challenge, the researchers in this study investigated how to automatically generate pathology reports from WSIs.

First, they built the largest dataset of high-quality WSI-text pairs, called PathText, by collecting and cleaning pathology reports that describe diagnostic slides from the Cancer Genome Atlas (TCGA) project. This dataset provides the necessary training data for developing AI models to generate pathology reports.

Next, the researchers proposed a new multiple instance generative model (MI-Gen) that can produce pathology reports for gigapixel WSIs. This model learns to associate visual patterns in the WSIs with the corresponding text in the pathology reports.

The researchers benchmarked their MI-Gen model on a large subset of the TCGA-PathText dataset. The results show that their model can generate pathology reports containing multiple clinically relevant findings. Additionally, they found that simple extraction of information from the generated reports can achieve state-of-the-art performance on certain cancer subtyping tasks, surpassing previous approaches.

Technical Explanation

The researchers curated the largest WSI-text dataset, called PathText, by recognizing and cleaning pathology reports that describe diagnostic slides from the TCGA project. This dataset contains nearly 10,000 high-quality WSI-text pairs, providing the necessary training data for developing AI models to generate pathology reports.

To address the challenge of producing pathology reports from WSIs, the researchers proposed a multiple instance generative model (MI-Gen). This model learns to associate visual patterns in the WSIs with the corresponding text in the pathology reports. The MI-Gen architecture leverages both local and global information from the WSIs to generate coherent and clinically relevant pathology reports.

The researchers benchmarked their MI-Gen model on a large subset of the TCGA-PathText dataset. Experimental results show that their model can generate pathology reports containing multiple clinical clues and achieve competitive performance on certain slide-level tasks, such as breast cancer subtyping. Interestingly, they found that simple semantic extraction from the generated pathology reports can outperform previous state-of-the-art approaches on BRCA subtyping, achieving an F1 score of 0.838.

Critical Analysis

The researchers acknowledge several limitations of their study. First, the dataset they curated, PathText, may not capture the full diversity of pathology reports, as it is derived from a single source (TCGA). Expanding the dataset to include reports from multiple institutions could improve the generalizability of their model.

Additionally, the researchers do not provide a detailed analysis of the types of clinical clues and insights contained in the generated pathology reports. A more in-depth evaluation of the report quality and clinical relevance would help assess the practical utility of their approach.

Further research is also needed to understand the interpretability and explainability of the MI-Gen model's decision-making process. Techniques like interpretable machine learning or self-supervised learning could shed light on how the model associates visual patterns with textual descriptions, which would be valuable for building trust in the system.

Conclusion

This research presents a novel approach to automatically generate pathology reports from whole slide images, a critical task in digital pathology. By curating the largest WSI-text dataset, PathText, and developing the multiple instance generative model (MI-Gen), the researchers have made significant progress towards reducing the workload and improving the accuracy of pathology report writing.

The ability to generate clinically relevant pathology reports from WSIs has the potential to revolutionize the field of digital pathology, enabling faster and more consistent diagnoses, and ultimately leading to improved patient outcomes. While further research is needed to refine and validate the approach, this work represents an important step towards the goal of automating pathology report generation.