3D-MoE: A Mixture-of-Experts Multi-modal LLM for 3D Vision and Pose Diffusion via Rectified Flow
Journal:
arXiv
Published Date:
Jan 28, 2025
Abstract
3D vision and spatial reasoning have long been recognized as preferable for
accurately perceiving our three-dimensional world, especially when compared
with traditional visual reasoning based on 2D images. Due to the difficulties
in collecting high-quality 3D data, research in this area has only recently
gained momentum. With the advent of powerful large language models (LLMs),
multi-modal LLMs for 3D vision have been developed over the past few years.
However, most of these models focus primarily on the vision encoder for 3D
data. In this paper, we propose converting existing densely activated LLMs into
mixture-of-experts (MoE) models, which have proven effective for multi-modal
data processing. In addition to leveraging these models' instruction-following
capabilities, we further enable embodied task planning by attaching a diffusion
head, Pose-DiT, that employs a novel rectified flow diffusion scheduler.
Experimental results on 3D question answering and task-planning tasks
demonstrate that our 3D-MoE framework achieves improved performance with fewer
activated parameters.
