Time Series Representations for Classification Lie Hidden in Pretrained Vision Transformers
Journal:
arXiv
Published Date:
Jun 10, 2025
Abstract
Time series classification is a fundamental task in healthcare and industry,
yet the development of time series foundation models (TSFMs) remains limited by
the scarcity of publicly available time series datasets. In this work, we
propose Time Vision Transformer (TiViT), a framework that converts time series
into images to leverage the representational power of frozen Vision
Transformers (ViTs) pretrained on large-scale image datasets. First, we
theoretically motivate our approach by analyzing the 2D patching of ViTs for
time series, showing that it can increase the number of label-relevant tokens
and reduce the sample complexity. Second, we empirically demonstrate that TiViT
achieves state-of-the-art performance on standard time series classification
benchmarks by utilizing the hidden representations of large OpenCLIP models. We
explore the structure of TiViT representations and find that intermediate
layers with high intrinsic dimension are the most effective for time series
classification. Finally, we assess the alignment between TiViT and TSFM
representation spaces and identify a strong complementarity, with further
performance gains achieved by combining their features. Our findings reveal yet
another direction for reusing vision representations in a non-visual domain.
