Martian World Models: Controllable Video Synthesis with Physically Accurate 3D Reconstructions
Journal:
arXiv
Published Date:
Jul 10, 2025
Abstract
Synthesizing realistic Martian landscape videos is crucial for mission
rehearsal and robotic simulation. However, this task poses unique challenges
due to the scarcity of high-quality Martian data and the significant domain gap
between Martian and terrestrial imagery. To address these challenges, we
propose a holistic solution composed of two key components: 1) A data curation
pipeline Multimodal Mars Synthesis (M3arsSynth), which reconstructs 3D Martian
environments from real stereo navigation images, sourced from NASA's Planetary
Data System (PDS), and renders high-fidelity multiview 3D video sequences. 2) A
Martian terrain video generator, MarsGen, which synthesizes novel videos
visually realistic and geometrically consistent with the 3D structure encoded
in the data. Our M3arsSynth engine spans a wide range of Martian terrains and
acquisition dates, enabling the generation of physically accurate 3D surface
models at metric-scale resolution. MarsGen, fine-tuned on M3arsSynth data,
synthesizes videos conditioned on an initial image frame and, optionally,
camera trajectories or textual prompts, allowing for video generation in novel
environments. Experimental results show that our approach outperforms video
synthesis models trained on terrestrial datasets, achieving superior visual
fidelity and 3D structural consistency.
