DART: Differentiable Dynamic Adaptive Region Tokenizer for Vision Transformer and Mamba
Journal:
arXiv
Published Date:
Jun 12, 2025
Abstract
Recently, non-convolutional models such as the Vision Transformer (ViT) and
Vision Mamba (Vim) have achieved remarkable performance in computer vision
tasks. However, their reliance on fixed-size patches often results in excessive
encoding of background regions and omission of critical local details,
especially when informative objects are sparsely distributed. To address this,
we introduce a fully differentiable Dynamic Adaptive Region Tokenizer (DART),
which adaptively partitions images into content-dependent patches of varying
sizes. DART combines learnable region scores with piecewise differentiable
quantile operations to allocate denser tokens to information-rich areas.
Despite introducing only approximately 1 million (1M) additional parameters,
DART improves accuracy by 2.1% on DeiT (ImageNet-1K). Unlike methods that
uniformly increase token density to capture fine-grained details, DART offers a
more efficient alternative, achieving 45% FLOPs reduction with superior
performance. Extensive experiments on DeiT, Vim, and VideoMamba confirm that
DART consistently enhances accuracy while incurring minimal or even reduced
computational overhead. Code is available at
https://github.com/HCPLab-SYSU/DART.
