GViT: Representing Images as Gaussians for Visual Recognition
Journal:
arXiv
Published Date:
Jun 30, 2025
Abstract
We introduce GVIT, a classification framework that abandons conventional
pixel or patch grid input representations in favor of a compact set of
learnable 2D Gaussians. Each image is encoded as a few hundred Gaussians whose
positions, scales, orientations, colors, and opacities are optimized jointly
with a ViT classifier trained on top of these representations. We reuse the
classifier gradients as constructive guidance, steering the Gaussians toward
class-salient regions while a differentiable renderer optimizes an image
reconstruction loss. We demonstrate that by 2D Gaussian input representations
coupled with our GVIT guidance, using a relatively standard ViT architecture,
closely matches the performance of a traditional patch-based ViT, reaching a
76.9% top-1 accuracy on Imagenet-1k using a ViT-B architecture.
