Predicting Genetic Mutation from Whole Slide Images via Biomedical-Linguistic Knowledge Enhanced Multi-label Classification

Read original: arXiv:2406.02990 - Published 6/6/2024 by Gexin Huang, Chenfei Wu, Mingjie Li, Xiaojun Chang, Ling Chen, Ying Sun, Shen Zhao, Xiaodan Liang, Liang Lin

🏷️

Overview

This paper presents a method for predicting genetic mutations from whole slide images (WSI) of cancer tissue samples.
The approach combines biomedical and linguistic knowledge to enhance a multi-label classification model for identifying genetic alterations from pathology images.
The research aims to improve cancer diagnosis and treatment by automating the process of detecting genetic mutations from digitized tissue samples.

Plain English Explanation

Detecting genetic mutations in cancer cells is crucial for understanding the disease and guiding treatment. However, manually reviewing tissue samples under a microscope to identify these genetic changes is a time-consuming and labor-intensive process. This paper introduces a new technique to automate this task using artificial intelligence.

The researchers developed a machine learning model that can analyze whole slide images of cancer tissue and predict the genetic mutations present. To do this, they incorporated two key innovations:

Biomedical Knowledge: The model was trained on a large database of information about the genetic drivers of cancer and how they relate to tissue morphology. This "biomedical knowledge" helps the AI system understand the connections between image features and underlying genetic alterations.
Linguistic Knowledge: The researchers also incorporated natural language processing techniques to leverage the wealth of biomedical literature describing the relationships between genetics and pathology. This "linguistic knowledge" further enhances the model's ability to interpret the visual patterns in the tissue samples.

By combining these biomedical and linguistic insights, the researchers were able to create a more powerful and accurate system for predicting genetic mutations from pathology images. This advance could significantly streamline cancer diagnosis and treatment planning, potentially leading to better outcomes for patients.

Technical Explanation

The core of the paper's approach is a multi-label classification model that can simultaneously identify multiple genetic mutations from whole slide images of tumor tissue. To enhance the performance of this model, the researchers incorporated two key innovations:

Biomedical Knowledge Integration: The model was pretrained on a large dataset of annotated pathology images and associated genomic data. This allowed the model to learn the visual patterns in the tissue samples that are indicative of specific genetic alterations, leveraging the biomedical knowledge embedded in the training data.
Linguistic Knowledge Integration: In addition, the researchers used natural language processing techniques to extract relevant biomedical knowledge from the scientific literature. This "linguistic knowledge" was then used to further refine the model's understanding of the relationships between image features and genetic mutations, improving its slide representation learning capabilities.

The resulting hybrid machine learning model was evaluated on a large dataset of WSIs and associated genomic profiles. The experiments demonstrated that the integration of biomedical and linguistic knowledge led to significant improvements in the model's ability to accurately predict genetic mutations from the pathology images.

Critical Analysis

The researchers acknowledge several limitations and areas for future work. First, the model was trained and evaluated on a cohort of Chinese patients, so its generalizability to other populations remains to be seen. Additionally, the study focused on a predefined set of genetic alterations, whereas in practice, clinicians may need to identify a broader range of mutations.

Another potential issue is the reliance on the quality and completeness of the underlying biomedical and linguistic knowledge bases. If these resources contain biases or gaps, they could negatively impact the model's performance. Further research is needed to better understand the model's failure modes and robustness to noisy or incomplete data.

Finally, while the paper demonstrates impressive results, the clinical utility of the approach will depend on its ability to scale to real-world diagnostic settings. Factors such as workflow integration, interpretability, and regulatory approval will all be important considerations for translating this research into practical applications.

Conclusion

This paper presents a novel approach for predicting genetic mutations from whole slide images of cancer tissue. By integrating biomedical and linguistic knowledge into a multi-label classification model, the researchers were able to significantly improve the accuracy of genetic mutation detection from pathology images.

The potential impact of this work is significant, as it could streamline the process of cancer diagnosis and treatment planning. By automating the identification of key genetic drivers, this technology could help clinicians make more informed decisions and deliver more personalized care to patients.

While further research is needed to address the limitations and scale the approach to real-world settings, this paper represents an important step forward in the convergence of computational pathology and genomics. As these fields continue to advance, we can expect to see increasingly powerful AI-based tools that enhance our understanding of cancer and improve patient outcomes.

This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!

Follow @aimodelsfyi on 𝕏 →

Related Papers

🏷️

Predicting Genetic Mutation from Whole Slide Images via Biomedical-Linguistic Knowledge Enhanced Multi-label Classification

Gexin Huang, Chenfei Wu, Mingjie Li, Xiaojun Chang, Ling Chen, Ying Sun, Shen Zhao, Xiaodan Liang, Liang Lin

Predicting genetic mutations from whole slide images is indispensable for cancer diagnosis. However, existing work training multiple binary classification models faces two challenges: (a) Training multiple binary classifiers is inefficient and would inevitably lead to a class imbalance problem. (b) The biological relationships among genes are overlooked, which limits the prediction performance. To tackle these challenges, we innovatively design a Biological-knowledge enhanced PathGenomic multi-label Transformer to improve genetic mutation prediction performances. BPGT first establishes a novel gene encoder that constructs gene priors by two carefully designed modules: (a) A gene graph whose node features are the genes' linguistic descriptions and the cancer phenotype, with edges modeled by genes' pathway associations and mutation consistencies. (b) A knowledge association module that fuses linguistic and biomedical knowledge into gene priors by transformer-based graph representation learning, capturing the intrinsic relationships between different genes' mutations. BPGT then designs a label decoder that finally performs genetic mutation prediction by two tailored modules: (a) A modality fusion module that firstly fuses the gene priors with critical regions in WSIs and obtains gene-wise mutation logits. (b) A comparative multi-label loss that emphasizes the inherent comparisons among mutation status to enhance the discrimination capabilities. Sufficient experiments on The Cancer Genome Atlas benchmark demonstrate that BPGT outperforms the state-of-the-art.

6/6/2024

Prompting Whole Slide Image Based Genetic Biomarker Prediction

Ling Zhang, Boxiang Yun, Xingran Xie, Qingli Li, Xinxing Li, Yan Wang

Prediction of genetic biomarkers, e.g., microsatellite instability and BRAF in colorectal cancer is crucial for clinical decision making. In this paper, we propose a whole slide image (WSI) based genetic biomarker prediction method via prompting techniques. Our work aims at addressing the following challenges: (1) extracting foreground instances related to genetic biomarkers from gigapixel WSIs, and (2) the interaction among the fine-grained pathological components in WSIs.Specifically, we leverage large language models to generate medical prompts that serve as prior knowledge in extracting instances associated with genetic biomarkers. We adopt a coarse-to-fine approach to mine biomarker information within the tumor microenvironment. This involves extracting instances related to genetic biomarkers using coarse medical prior knowledge, grouping pathology instances into fine-grained pathological components and mining their interactions. Experimental results on two colorectal cancer datasets show the superiority of our method, achieving 91.49% in AUC for MSI classification. The analysis further shows the clinical interpretability of our method. Code is publicly available at https://github.com/DeepMed-Lab-ECNU/PromptBio.

7/16/2024

Pathology-genomic fusion via biologically informed cross-modality graph learning for survival analysis

Zeyu Zhang, Yuanshen Zhao, Jingxian Duan, Yaou Liu, Hairong Zheng, Dong Liang, Zhenyu Zhang, Zhi-Cheng Li

The diagnosis and prognosis of cancer are typically based on multi-modal clinical data, including histology images and genomic data, due to the complex pathogenesis and high heterogeneity. Despite the advancements in digital pathology and high-throughput genome sequencing, establishing effective multi-modal fusion models for survival prediction and revealing the potential association between histopathology and transcriptomics remains challenging. In this paper, we propose Pathology-Genome Heterogeneous Graph (PGHG) that integrates whole slide images (WSI) and bulk RNA-Seq expression data with heterogeneous graph neural network for cancer survival analysis. The PGHG consists of biological knowledge-guided representation learning network and pathology-genome heterogeneous graph. The representation learning network utilizes the biological prior knowledge of intra-modal and inter-modal data associations to guide the feature extraction. The node features of each modality are updated through attention-based graph learning strategy. Unimodal features and bi-modal fused features are extracted via attention pooling module and then used for survival prediction. We evaluate the model on low-grade gliomas, glioblastoma, and kidney renal papillary cell carcinoma datasets from the Cancer Genome Atlas (TCGA) and the First Affiliated Hospital of Zhengzhou University (FAHZU). Extensive experimental results demonstrate that the proposed method outperforms both unimodal and other multi-modal fusion models. For demonstrating the model interpretability, we also visualize the attention heatmap of pathological images and utilize integrated gradient algorithm to identify important tissue structure, biological pathways and key genes.

4/15/2024

🔮

Efficient and generalizable prediction of molecular alterations in multiple cancer cohorts using H&E whole slide images

Kshitij Ingale, Sun Hae Hong, Qiyuan Hu, Renyu Zhang, Bo Osinski, Mina Khoshdeli, Josh Och, Kunal Nagpal, Martin C. Stumpe, Rohan P. Joshi

Molecular testing of tumor samples for targetable biomarkers is restricted by a lack of standardization, turnaround-time, cost, and tissue availability across cancer types. Additionally, targetable alterations of low prevalence may not be tested in routine workflows. Algorithms that predict DNA alterations from routinely generated hematoxylin and eosin (H&E)-stained images could prioritize samples for confirmatory molecular testing. Costs and the necessity of a large number of samples containing mutations limit approaches that train individual algorithms for each alteration. In this work, models were trained for simultaneous prediction of multiple DNA alterations from H&E images using a multi-task approach. Compared to biomarker-specific models, this approach performed better on average, with pronounced gains for rare mutations. The models reasonably generalized to independent temporal-holdout, externally-stained, and multi-site TCGA test sets. Additionally, whole slide image embeddings derived using multi-task models demonstrated strong performance in downstream tasks that were not a part of training. Overall, this is a promising approach to develop clinically useful algorithms that provide multiple actionable predictions from a single slide.

7/23/2024