A Pragmatic Note on Evaluating Generative Models with Fréchet Inception Distance for Retinal Image Synthesis
Journal:
arXiv
Published Date:
Feb 24, 2025
Abstract
Fr\'echet Inception Distance (FID), computed with an ImageNet pretrained
Inception-v3 network, is widely used as a state-of-the-art evaluation metric
for generative models. It assumes that feature vectors from Inception-v3 follow
a multivariate Gaussian distribution and calculates the 2-Wasserstein distance
based on their means and covariances. While FID effectively measures how
closely synthetic data match real data in many image synthesis tasks, the
primary goal in biomedical generative models is often to enrich training
datasets ideally with corresponding annotations. For this purpose, the gold
standard for evaluating generative models is to incorporate synthetic data into
downstream task training, such as classification and segmentation, to
pragmatically assess its performance. In this paper, we examine cases from
retinal imaging modalities, including color fundus photography and optical
coherence tomography, where FID and its related metrics misalign with
task-specific evaluation goals in classification and segmentation. We highlight
the limitations of using various metrics, represented by FID and its variants,
as evaluation criteria for these applications and address their potential
caveats in broader biomedical imaging modalities and downstream tasks.
