Inverse receptive field attention for naturalistic image reconstruction from the brain
Journal:
arXiv
Published Date:
Jan 6, 2025
Abstract
Visual perception in the brain largely depends on the organization of
neuronal receptive fields. Although extensive research has delineated the
coding principles of receptive fields, most studies have been constrained by
their foundational assumptions. Moreover, while machine learning has
successfully been used to reconstruct images from brain data, this approach
faces significant challenges, including inherent feature biases in the model
and the complexities of brain structure and function. In this study, we
introduce an inverse receptive field attention (IRFA) model, designed to
reconstruct naturalistic images from neurophysiological data in an end-to-end
fashion. This approach aims to elucidate the tuning properties and
representational transformations within the visual cortex. The IRFA model
incorporates an attention mechanism that determines the inverse receptive field
for each pixel, weighting neuronal responses across the visual field and
feature spaces. This method allows for an examination of the dynamics of
neuronal representations across stimuli in both spatial and feature dimensions.
Our results show highly accurate reconstructions of naturalistic data,
independent of pre-trained models. Notably, IRF models trained on macaque V1,
V4, and IT regions yield remarkably consistent spatial receptive fields across
different stimuli, while the features to which neuronal representations are
selective exhibit significant variation. Additionally, we propose a data-driven
method to explore representational clustering within various visual areas,
further providing testable hypotheses.
