Zero-Shot Whole Slide Image Retrieval in Histopathology Using Embeddings of Foundation Models

Overview

• Researchers have tested recently published foundation models for histopathology image retrieval.
• They report macro average of F1 score for top-1, top-3, and top-5 retrievals.
• They perform zero-shot retrievals without altering embeddings or training any classifier.
• The test data is from The Cancer Genome Atlas (TCGA), consisting of 23 organs and 117 cancer subtypes.
• They used the Yottixel platform to perform whole slide image (WSI) search using patches.
• The achieved F1 scores show low performance, e.g., for top-5 retrievals: 27% ± 13% (Yottixel-DenseNet), 42% ± 14% (Yottixel-UNI), 40% ± 13% (Yottixel-Virchow), and 41% ± 13% (Yottixel-GigaPath).
• The results for GigaPath WSI will be delayed due to significant computational resources required.

Plain English Explanation

The researchers have tested recently developed foundation models for their ability to retrieve relevant images from a large dataset of medical images, specifically histopathology slides. They did this without needing to fine-tune or train any new models, a process called "zero-shot" retrieval.

The test dataset consisted of diagnostic slides from the The Cancer Genome Atlas (TCGA), which includes images from 23 different organs and 117 different types of cancer. The researchers used a platform called Yottixel to search through these whole slide images using smaller image "patches."

The researchers measured the performance of the models using a metric called the F1 score, which considers both precision (how many of the retrieved images were relevant) and recall (how many of the relevant images were retrieved). They looked at the top-1, top-3, and top-5 retrievals, meaning the first, first three, and first five images retrieved by the models.

The results showed that the models generally had low performance, with top-5 F1 scores ranging from 27% to 42%. This suggests that while these foundation models can be used for histopathology image retrieval, they still have significant room for improvement. The researchers also noted that processing the GigaPath whole slide images required significant computational resources, so those results will be delayed.

Technical Explanation

The researchers evaluated the performance of recently published foundation models for the task of histopathology image retrieval. They used a zero-shot approach, meaning they did not fine-tune or train any new models, but rather used the pre-trained embeddings directly.

As test data, the researchers used diagnostic slides from The Cancer Genome Atlas (TCGA), which consists of 23 organs and 117 cancer subtypes. They used the Yottixel platform to perform whole slide image (WSI) search using image patches.

The researchers report the macro average of F1 score for top-1, majority of top-3, and majority of top-5 retrievals. The F1 score is a metric that considers both precision (how many of the retrieved images were relevant) and recall (how many of the relevant images were retrieved).

The results show relatively low performance, with top-5 F1 scores of 27% ± 13% (Yottixel-DenseNet), 42% ± 14% (Yottixel-UNI), 40% ± 13% (Yottixel-Virchow), and 41% ± 13% (Yottixel-GigaPath). The researchers note that the results for the GigaPath WSI will be delayed due to the significant computational resources required for processing.

Critical Analysis

The researchers acknowledge the low performance of the tested foundation models for histopathology image retrieval, as evidenced by the relatively low F1 scores, particularly for the top-1 and top-3 retrievals. This suggests that while these models can be used for this task, they still have significant room for improvement.

One potential limitation of the study is the use of a single test dataset, the TCGA, which may not be representative of all histopathology images. Additionally, the researchers note the computational challenges of processing the GigaPath whole slide images, which could limit the broader applicability of the tested models.

Further research could explore ways to improve the performance of these foundation models, such as fine-tuning on larger or more diverse histopathology datasets, or investigating the use of multimodal approaches that incorporate additional data sources beyond just the images.

Conclusion

The researchers have evaluated the performance of recently published foundation models for the task of histopathology image retrieval using a zero-shot approach. The results show relatively low performance, with top-5 F1 scores ranging from 27% to 42%. While these models can be used for this task, the findings suggest that significant improvements are still needed to make them more effective for real-world clinical applications in computational pathology.