Instrument-Splatting: Controllable Photorealistic Reconstruction of Surgical Instruments Using Gaussian Splatting
Journal:
arXiv
Published Date:
Mar 6, 2025
Abstract
Real2Sim is becoming increasingly important with the rapid development of
surgical artificial intelligence (AI) and autonomy. In this work, we propose a
novel Real2Sim methodology, Instrument-Splatting, that leverages 3D Gaussian
Splatting to provide fully controllable 3D reconstruction of surgical
instruments from monocular surgical videos. To maintain both high visual
fidelity and manipulability, we introduce a geometry pre-training to bind
Gaussian point clouds on part mesh with accurate geometric priors and define a
forward kinematics to control the Gaussians as flexible as real instruments.
Afterward, to handle unposed videos, we design a novel instrument pose tracking
method leveraging semantics-embedded Gaussians to robustly refine per-frame
instrument poses and joint states in a render-and-compare manner, which allows
our instrument Gaussian to accurately learn textures and reach photorealistic
rendering. We validated our method on 2 publicly released surgical videos and 4
videos collected on ex vivo tissues and green screens. Quantitative and
qualitative evaluations demonstrate the effectiveness and superiority of the
proposed method.
