Understanding Performance Plateaus and Representation Collapse in 3D MRI Transformers Trained to Classify Brain Disease

In medical imaging AI, we commonly assume that diagnostic classifiers will increase in accuracy if they are trained on more data, but in reality, their performance often plateaus. To understand this behavior, we ran a spectral analysis of 3D vision transformers, a common supervised deep learning model that can be trained to classify disease based on 3D brain MRI. For simpler classification methods such as linear discriminant analysis, the classifier performance AUC(N), when trained on a sample of size N, is known to depend on the encoder's eigenspectrum. Each class-discriminative direction in the latent space contributes only after the sample size N reaches a threshold, so the spectrum influences how performance scales with training data. Building on this theory, we show that supervised transformers compress almost all of their discriminative signal into one principal component (EVR@1 = 0.90--0.96): after this representation collapse happens, no further directions cross the threshold as training data increases, and accuracy plateaus. In principle, this plateau in performance could be due to either the encoder's learned feature space or the difficulty of the task. To distinguish these cases, we trained three types of supervised 3D transformers (ViT3D, MINiT, NIT) and, as a non-collapsing alternative, a frozen self-supervised encoder (DINO), to classify people with dementia or mild cognitive impairment (MCI) versus cognitively normal (CN) individuals from 3D brain MRI, using data from the Alzheimer's Disease Neuroimaging Initiative (ADNI). We froze each encoder and applied it to a new task, so that only the task difficulty varied. An easy task encoder (Dementia vs CN), transferred to the harder task (MCI vs CN), did not reproduce the plateau (MINiT threshold 9.2), but full retraining did (2758.5). The plateau is therefore due to the learned encoder, rather than task difficulty. Ranking the feature directions by the proportion of the disease signal they contain, we were able to identify a sign of representation collapse, i.e., a single dominant spike in the latent feature eigenspectrum. This spike was 11--26 times larger than predicted from the rest of the spectrum for the disease-trained (supervised) encoders, but was absent for DINO (0.66). To predict how transformer accuracy scales with the number of training scans, a large spike in the encoder spectrum suggests that the representation has collapsed and additional scans will not help; a small spike means that the signal is spread across many encoder latent space directions, and more training data should boost the classifier performance.