Image Recognition with Online Lightweight Vision Transformer: A Survey
Journal:
arXiv
Published Date:
May 6, 2025
Abstract
The Transformer architecture has achieved significant success in natural
language processing, motivating its adaptation to computer vision tasks. Unlike
convolutional neural networks, vision transformers inherently capture
long-range dependencies and enable parallel processing, yet lack inductive
biases and efficiency benefits, facing significant computational and memory
challenges that limit its real-world applicability. This paper surveys various
online strategies for generating lightweight vision transformers for image
recognition, focusing on three key areas: Efficient Component Design, Dynamic
Network, and Knowledge Distillation. We evaluate the relevant exploration for
each topic on the ImageNet-1K benchmark, analyzing trade-offs among precision,
parameters, throughput, and more to highlight their respective advantages,
disadvantages, and flexibility. Finally, we propose future research directions
and potential challenges in the lightweighting of vision transformers with the
aim of inspiring further exploration and providing practical guidance for the
community. Project Page: https://github.com/ajxklo/Lightweight-VIT
