Recognize Any Surgical Object: Unleashing the Power of Weakly-Supervised Data
Journal:
arXiv
Published Date:
Jan 25, 2025
Abstract
We present RASO, a foundation model designed to Recognize Any Surgical
Object, offering robust open-set recognition capabilities across a broad range
of surgical procedures and object classes, in both surgical images and videos.
RASO leverages a novel weakly-supervised learning framework that generates
tag-image-text pairs automatically from large-scale unannotated surgical
lecture videos, significantly reducing the need for manual annotations. Our
scalable data generation pipeline gathers 2,200 surgical procedures and
produces 3.6 million tag annotations across 2,066 unique surgical tags. Our
experiments show that RASO achieves improvements of 2.9 mAP, 4.5 mAP, 10.6 mAP,
and 7.2 mAP on four standard surgical benchmarks, respectively, in zero-shot
settings, and surpasses state-of-the-art models in supervised surgical action
recognition tasks. Code, model, and demo are available at
https://ntlm1686.github.io/raso.
