Information-Guided Diffusion Sampling for Dataset Distillation
Journal:
arXiv
Published Date:
Jul 7, 2025
Abstract
Dataset distillation aims to create a compact dataset that retains essential
information while maintaining model performance. Diffusion models (DMs) have
shown promise for this task but struggle in low images-per-class (IPC)
settings, where generated samples lack diversity. In this paper, we address
this issue from an information-theoretic perspective by identifying two key
types of information that a distilled dataset must preserve: ($i$) prototype
information $\mathrm{I}(X;Y)$, which captures label-relevant features; and
($ii$) contextual information $\mathrm{H}(X | Y)$, which preserves intra-class
variability. Here, $(X,Y)$ represents the pair of random variables
corresponding to the input data and its ground truth label, respectively.
Observing that the required contextual information scales with IPC, we propose
maximizing $\mathrm{I}(X;Y) + \beta \mathrm{H}(X | Y)$ during the DM sampling
process, where $\beta$ is IPC-dependent. Since directly computing
$\mathrm{I}(X;Y)$ and $\mathrm{H}(X | Y)$ is intractable, we develop
variational estimations to tightly lower-bound these quantities via a
data-driven approach. Our approach, information-guided diffusion sampling
(IGDS), seamlessly integrates with diffusion models and improves dataset
distillation across all IPC settings. Experiments on Tiny ImageNet and ImageNet
subsets show that IGDS significantly outperforms existing methods, particularly
in low-IPC regimes. The code will be released upon acceptance.
