EventGPT: Event Stream Understanding with Multimodal Large Language Models
Journal:
arXiv
Published Date:
Dec 1, 2024
Abstract
Event cameras record visual information as asynchronous pixel change streams,
excelling at scene perception under unsatisfactory lighting or high-dynamic
conditions. Existing multimodal large language models (MLLMs) concentrate on
natural RGB images, failing in scenarios where event data fits better. In this
paper, we introduce EventGPT, the first MLLM for event stream understanding, to
the best of our knowledge, marking a pioneering attempt to integrate large
language models (LLMs) with event stream comprehension. To mitigate the huge
domain gaps, we develop a three-stage optimization paradigm to gradually equip
a pre-trained LLM with the capability of understanding event-based scenes. Our
EventGPT comprises an event encoder, followed by a spatio-temporal aggregator,
a linear projector, an event-language adapter, and an LLM. Firstly, RGB
image-text pairs generated by GPT are leveraged to warm up the linear
projector, referring to LLaVA, as the gap between natural image and language
modalities is relatively smaller. Secondly, we construct a synthetic yet large
dataset, N-ImageNet-Chat, consisting of event frames and corresponding texts to
enable the use of the spatio-temporal aggregator and to train the
event-language adapter, thereby aligning event features more closely with the
language space. Finally, we gather an instruction dataset, Event-Chat, which
contains extensive real-world data to fine-tune the entire model, further
enhancing its generalization ability. We construct a comprehensive benchmark,
and experiments show that EventGPT surpasses previous state-of-the-art MLLMs in
generation quality, descriptive accuracy, and reasoning capability.
