NLP4Neuro: Sequence-to-sequence learning for neural population decoding
Journal:
arXiv
Published Date:
Jul 3, 2025
Abstract
Delineating how animal behavior arises from neural activity is a foundational
goal of neuroscience. However, as the computations underlying behavior unfold
in networks of thousands of individual neurons across the entire brain, this
presents challenges for investigating neural roles and computational mechanisms
in large, densely wired mammalian brains during behavior. Transformers, the
backbones of modern large language models (LLMs), have become powerful tools
for neural decoding from smaller neural populations. These modern LLMs have
benefited from extensive pre-training, and their sequence-to-sequence learning
has been shown to generalize to novel tasks and data modalities, which may also
confer advantages for neural decoding from larger, brain-wide activity
recordings. Here, we present a systematic evaluation of off-the-shelf LLMs to
decode behavior from brain-wide populations, termed NLP4Neuro, which we used to
test LLMs on simultaneous calcium imaging and behavior recordings in larval
zebrafish exposed to visual motion stimuli. Through NLP4Neuro, we found that
LLMs become better at neural decoding when they use pre-trained weights learned
from textual natural language data. Moreover, we found that a recent
mixture-of-experts LLM, DeepSeek Coder-7b, significantly improved behavioral
decoding accuracy, predicted tail movements over long timescales, and provided
anatomically consistent highly interpretable readouts of neuron salience.
NLP4Neuro demonstrates that LLMs are highly capable of informing brain-wide
neural circuit dissection.
