Scenario Dreamer: Vectorized Latent Diffusion for Generating Driving Simulation Environments
Journal:
arXiv
Published Date:
Mar 28, 2025
Abstract
We introduce Scenario Dreamer, a fully data-driven generative simulator for
autonomous vehicle planning that generates both the initial traffic scene -
comprising a lane graph and agent bounding boxes - and closed-loop agent
behaviours. Existing methods for generating driving simulation environments
encode the initial traffic scene as a rasterized image and, as such, require
parameter-heavy networks that perform unnecessary computation due to many empty
pixels in the rasterized scene. Moreover, we find that existing methods that
employ rule-based agent behaviours lack diversity and realism. Scenario Dreamer
instead employs a novel vectorized latent diffusion model for initial scene
generation that directly operates on the vectorized scene elements and an
autoregressive Transformer for data-driven agent behaviour simulation. Scenario
Dreamer additionally supports scene extrapolation via diffusion inpainting,
enabling the generation of unbounded simulation environments. Extensive
experiments show that Scenario Dreamer outperforms existing generative
simulators in realism and efficiency: the vectorized scene-generation base
model achieves superior generation quality with around 2x fewer parameters, 6x
lower generation latency, and 10x fewer GPU training hours compared to the
strongest baseline. We confirm its practical utility by showing that
reinforcement learning planning agents are more challenged in Scenario Dreamer
environments than traditional non-generative simulation environments,
especially on long and adversarial driving environments.
