Efficient Multi-Camera Tokenization with Triplanes for End-to-End Driving
Journal:
arXiv
Published Date:
Jun 13, 2025
Abstract
Autoregressive Transformers are increasingly being deployed as end-to-end
robot and autonomous vehicle (AV) policy architectures, owing to their
scalability and potential to leverage internet-scale pretraining for
generalization. Accordingly, tokenizing sensor data efficiently is paramount to
ensuring the real-time feasibility of such architectures on embedded hardware.
To this end, we present an efficient triplane-based multi-camera tokenization
strategy that leverages recent advances in 3D neural reconstruction and
rendering to produce sensor tokens that are agnostic to the number of input
cameras and their resolution, while explicitly accounting for their geometry
around an AV. Experiments on a large-scale AV dataset and state-of-the-art
neural simulator demonstrate that our approach yields significant savings over
current image patch-based tokenization strategies, producing up to 72% fewer
tokens, resulting in up to 50% faster policy inference while achieving the same
open-loop motion planning accuracy and improved offroad rates in closed-loop
driving simulations.
