SLEAP: A deep learning system for multi-animal pose tracking.

Journal: Nature methods
Published Date: Apr 4, 2022

Abstract

The desire to understand how the brain generates and patterns behavior has driven rapid methodological innovation in tools to quantify natural animal behavior. While advances in deep learning and computer vision have enabled markerless pose estimation in individual animals, extending these to multiple animals presents unique challenges for studies of social behaviors or animals in their natural environments. Here we present Social LEAP Estimates Animal Poses (SLEAP), a machine learning system for multi-animal pose tracking. This system enables versatile workflows for data labeling, model training and inference on previously unseen data. SLEAP features an accessible graphical user interface, a standardized data model, a reproducible configuration system, over 30 model architectures, two approaches to part grouping and two approaches to identity tracking. We applied SLEAP to seven datasets across flies, bees, mice and gerbils to systematically evaluate each approach and architecture, and we compare it with other existing approaches. SLEAP achieves greater accuracy and speeds of more than 800 frames per second, with latencies of less than 3.5 ms at full 1,024 × 1,024 image resolution. This makes SLEAP usable for real-time applications, which we demonstrate by controlling the behavior of one animal on the basis of the tracking and detection of social interactions with another animal.

Authors

  • Talmo D Pereira
    Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA.
  • Nathaniel Tabris
    Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA.
  • Arie Matsliah
    Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA.
  • David M Turner
    Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA.
  • Junyu Li
    School of Electrical and Mechanical Engineering, Hefei Technology College, Hefei, China.
  • Shruthi Ravindranath
    Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA.
  • Eleni S Papadoyannis
    Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA.
  • Edna Normand
    Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA.
  • David S Deutsch
    Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA.
  • Z Yan Wang
    Department of Ecology and Evolutionary Biology, Princeton University, Princeton, NJ, USA.
  • Grace C McKenzie-Smith
    Department of Physics, Princeton University, Princeton, NJ, USA.
  • Catalin C Mitelut
    Center for Neural Science, New York University, New York, NY, USA.
  • Marielisa Diez Castro
    Center for Neural Science, New York University, New York, NY, USA.
  • John D'Uva
    Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA.
  • Mikhail Kislin
    Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA.
  • Dan H Sanes
    Center for Neural Science, New York University, New York, NY, USA.
  • Sarah D Kocher
    Department of Ecology and Evolutionary Biology, Princeton University, Princeton, NJ, USA.
  • Samuel S-H Wang
    Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA.
  • Annegret L Falkner
    Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA.
  • Joshua W Shaevitz
    Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA. shaevitz@princeton.edu.
  • Mala Murthy
    Princeton Neuroscience Institute, Princeton University, Princeton, NJ, USA. mmurthy@princeton.edu.

Keywords

External Resources

Popular Topics
Recent Journals