VORTEX: Challenging CNNs at Texture Recognition by using Vision Transformers with Orderless and Randomized Token Encodings
Journal:
arXiv
Published Date:
Mar 9, 2025
Abstract
Texture recognition has recently been dominated by ImageNet-pre-trained deep
Convolutional Neural Networks (CNNs), with specialized modifications and
feature engineering required to achieve state-of-the-art (SOTA) performance.
However, although Vision Transformers (ViTs) were introduced a few years ago,
little is known about their texture recognition ability. Therefore, in this
work, we introduce VORTEX (ViTs with Orderless and Randomized Token Encodings
for Texture Recognition), a novel method that enables the effective use of ViTs
for texture analysis. VORTEX extracts multi-depth token embeddings from
pre-trained ViT backbones and employs a lightweight module to aggregate
hierarchical features and perform orderless encoding, obtaining a better image
representation for texture recognition tasks. This approach allows seamless
integration with any ViT with the common transformer architecture. Moreover, no
fine-tuning of the backbone is performed, since they are used only as frozen
feature extractors, and the features are fed to a linear SVM. We evaluate
VORTEX on nine diverse texture datasets, demonstrating its ability to achieve
or surpass SOTA performance in a variety of texture analysis scenarios. By
bridging the gap between texture recognition with CNNs and transformer-based
architectures, VORTEX paves the way for adopting emerging transformer
foundation models. Furthermore, VORTEX demonstrates robust computational
efficiency when coupled with ViT backbones compared to CNNs with similar costs.
The method implementation and experimental scripts are publicly available in
our online repository.
