Feature Extraction for Generative Medical Imaging Evaluation: New Evidence Against an Evolving Trend

Overview

• The paper examines the use of the Fréchet Inception Distance (FID) metric for assessing the quality of synthetic medical images generated by Generative Adversarial Networks (GANs).
• FID is a widely used metric for evaluating the quality of synthetic images, but it relies on an ImageNet-based feature extractor, which raises questions about its applicability to medical imaging.
• The study compares the performance of ImageNet-based and medical image-based (RadImageNet) feature extractors in computing FID for medical imaging tasks.

Plain English Explanation

The Fréchet Inception Distance (FID) is a popular way to evaluate the quality of synthetic images generated by AI models. It works by comparing the features of the synthetic images to the features of real images. The more similar the features, the higher the quality of the synthetic images.

However, the FID metric relies on a feature extractor trained on the ImageNet dataset, which is a general collection of everyday images. This raises the question of whether the FID metric is suitable for evaluating synthetic medical images, which can have very different visual characteristics compared to everyday images.

To address this, the researchers in this study tried using feature extractors trained on medical images (called RadImageNet) instead of the standard ImageNet-based extractor. They evaluated 16 different GAN models across four medical imaging modalities and four data augmentation techniques, using both ImageNet-based and RadImageNet-based FID scores.

The key finding is that the ImageNet-based FID scores were more consistent with human expert judgments of the synthetic medical image quality, compared to the RadImageNet-based scores. This was surprising, as the researchers had expected the medical image-trained extractor to perform better for this task.

Technical Explanation

The researchers evaluated 16 different StyleGAN2 networks trained on four medical imaging modalities (CT, MRI, X-Ray, and Histology) and four data augmentation techniques. They computed FID scores using eleven different feature extractors - some trained on ImageNet, and some trained on a medical image dataset called RadImageNet.

To assess the validity of the FID scores, the researchers conducted visual Turing tests, where human experts were asked to judge the quality of the synthetic medical images. They found that the FID scores derived from the ImageNet-trained SwAV feature extractor were significantly correlated with the expert evaluations, while the rankings based on RadImageNet-trained extractors were much more volatile and inconsistent with human judgment.

This suggests that, contrary to prevailing assumptions, medical image-trained feature extractors do not inherently improve the reliability of FID for medical imaging tasks. In fact, the researchers found that the ImageNet-based extractors produced FID scores that were more aligned with human perception of synthetic medical image quality.

Critical Analysis

The researchers acknowledge several limitations of their study. First, they only evaluated a single GAN architecture (StyleGAN2) and a limited set of medical imaging modalities and data augmentation techniques. It's possible that the findings may not generalize to other GAN models or a wider range of medical imaging applications.

Additionally, the researchers only used visual Turing tests to assess human judgment, which may not capture all aspects of image quality that are important for medical use cases. Further validation with domain experts performing clinically-relevant tasks could provide additional insights.

Another potential issue is the choice of the RadImageNet dataset for training the medical image-based feature extractors. While this dataset is larger and more diverse than many medical image datasets, it may still not be representative of the full range of medical imaging data and use cases.

Despite these limitations, the study provides valuable evidence challenging the common assumption that medical image-trained feature extractors are inherently superior for evaluating synthetic medical images using the FID metric. The researchers encourage further research to rethink perceptual metrics for medical image translation and to detect potential domain shifts when applying FID and other evaluation metrics across different medical imaging domains.

Conclusion

This study casts doubt on the prevailing assumption that adapting the FID metric to medical imaging by using feature extractors trained on medical image datasets will inherently improve its reliability and alignment with human judgment. The researchers found that ImageNet-based feature extractors actually produced FID scores that were more consistent with expert evaluations of synthetic medical image quality compared to their medical image-trained counterparts.

These findings suggest that the choice of feature extractor for computing FID is a critical factor, and that medical image-specific feature extractors do not necessarily outperform general-purpose extractors for this task. The study encourages further research to better understand the limitations of FID and other perceptual metrics when applied to medical imaging, and to explore alternative evaluation approaches that may be more suitable for this domain.