Shared functional specialization in transformer-based language models and the human brain.

Journal: Nature communications
Published Date: Jun 29, 2024

Abstract

When processing language, the brain is thought to deploy specialized computations to construct meaning from complex linguistic structures. Recently, artificial neural networks based on the Transformer architecture have revolutionized the field of natural language processing. Transformers integrate contextual information across words via structured circuit computations. Prior work has focused on the internal representations ("embeddings") generated by these circuits. In this paper, we instead analyze the circuit computations directly: we deconstruct these computations into the functionally-specialized "transformations" that integrate contextual information across words. Using functional MRI data acquired while participants listened to naturalistic stories, we first verify that the transformations account for considerable variance in brain activity across the cortical language network. We then demonstrate that the emergent computations performed by individual, functionally-specialized "attention heads" differentially predict brain activity in specific cortical regions. These heads fall along gradients corresponding to different layers and context lengths in a low-dimensional cortical space.

Authors

  • Sreejan Kumar
    Neuroscience Institute, Princeton University, Princeton, New Jersey, United States of America.
  • Theodore R Sumers
    Department of Computer Science, Princeton University, Princeton, NJ, 08540, USA. sumers@princeton.edu.
  • Takateru Yamakoshi
    Faculty of Medicine, The University of Tokyo, Bunkyo-ku, Tokyo, 113-0033, Japan.
  • Ariel Goldstein
    Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA.
  • Uri Hasson
    Princeton University, United States.
  • Kenneth A Norman
    Princeton Neuroscience Institute and Department of Psychology, Princeton, NJ 08544, USA.
  • Thomas L Griffiths
    Department of Psychology, University of California, Berkeley, USA.
  • Robert D Hawkins
    Princeton Neuroscience Institute, Princeton University, Princeton, NJ, 08540, USA.
  • Samuel A Nastase
    Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA.

Keywords

External Resources

Popular Topics
Recent Journals