Real-Time Segmentation and Classification of Birdsong Syllables for Learning Experiments.

Songbirds are essential for studying neuronal mechanisms of learned vocalizations. Closed-loop interventions require online recognition of a specific target syllable while the bird is singing, for example for manipulation of auditory feedback, song-triggered neuronal microstimulation or optogenetics. Existing tools for closed-loop interventions can recognize only single syllables through manually created templates, with limited flexibility to adapt to new experiments. We here present Moove (Marking Online using only the Onsets of Vocal Elements), a novel neural network approach to real-time syllable segmentation and classification of Bengalese finch songs. Moove's two-stage architecture detects syllable onsets and offsets, and classifies syllables using acoustic information only from the first part of the syllable, enabling precise temporal contingency between behavior and feedback. We verify Moove's fast and accurate online annotation of all recorded syllables in five adult male Bengalese finches (Lonchura striata domestica). To validate Moove as a tool for learning experiments, we trained one adult male Bengalese finch with an established protocol: a specific target syllable is covered with noise, which masks auditory feedback and leads the bird to introduce specific modifications to syllable sequencing. The trained bird learned to avoid the targeted syllable sequence with comparable outcomes to previous reinforcement learning experiments. Our results show that Moove can correctly segment and classify Bengalese finch syllables in real time, with speed and reliability that allows effective operant conditioning experiments. Moove could be used for other closed-loop experiments on vocal signals, making it a crucial tool for future investigations of birdsong sequencing.Significance Statement Moove (Marking Online using only the Onsets of Vocal Elements) is a new tool for the real-time annotation of birdsong. It annotates song syllables while they are still being sung, allowing experimental manipulations such as reinforcement, auditory feedback disruption, or stimulation to be applied to ongoing syllables. Moove uses a two-stage architecture with convolutional neural networks for segmentation and classification. We verify Moove's functionality in a reinforcement learning experiment by training a Bengalese finch to change its song sequence through disturbed auditory feedback. Moove is easy to use with a graphical user interface for network training and manual verification. Its open-source Python code can be adapted to a wide range of applications on vocal signals.