PlantSegNeRF: A few-shot, cross-dataset method for plant 3D instance point cloud reconstruction via joint-channel NeRF with multi-view image instance matching
Journal:
arXiv
Published Date:
Jul 1, 2025
Abstract
Organ segmentation of plant point clouds is a prerequisite for the
high-resolution and accurate extraction of organ-level phenotypic traits.
Although the fast development of deep learning has boosted much research on
segmentation of plant point clouds, the existing techniques for organ
segmentation still face limitations in resolution, segmentation accuracy, and
generalizability across various plant species. In this study, we proposed a
novel approach called plant segmentation neural radiance fields (PlantSegNeRF),
aiming to directly generate high-precision instance point clouds from
multi-view RGB image sequences for a wide range of plant species. PlantSegNeRF
performed 2D instance segmentation on the multi-view images to generate
instance masks for each organ with a corresponding ID. The multi-view instance
IDs corresponding to the same plant organ were then matched and refined using a
specially designed instance matching module. The instance NeRF was developed to
render an implicit scene, containing color, density, semantic and instance
information. The implicit scene was ultimately converted into high-precision
plant instance point clouds based on the volume density. The results proved
that in semantic segmentation of point clouds, PlantSegNeRF outperformed the
commonly used methods, demonstrating an average improvement of 16.1%, 18.3%,
17.8%, and 24.2% in precision, recall, F1-score, and IoU compared to the
second-best results on structurally complex datasets. More importantly,
PlantSegNeRF exhibited significant advantages in plant point cloud instance
segmentation tasks. Across all plant datasets, it achieved average improvements
of 11.7%, 38.2%, 32.2% and 25.3% in mPrec, mRec, mCov, mWCov, respectively.
This study extends the organ-level plant phenotyping and provides a
high-throughput way to supply high-quality 3D data for the development of
large-scale models in plant science.
