EIT-1M: One Million EEG-Image-Text Pairs for Human Visual-textual Recognition and More
Journal:
arXiv
Published Date:
Jul 2, 2024
Abstract
Recently, electroencephalography (EEG) signals have been actively
incorporated to decode brain activity to visual or textual stimuli and achieve
object recognition in multi-modal AI. Accordingly, endeavors have been focused
on building EEG-based datasets from visual or textual single-modal stimuli.
However, these datasets offer limited EEG epochs per category, and the complex
semantics of stimuli presented to participants compromise their quality and
fidelity in capturing precise brain activity. The study in neuroscience unveils
that the relationship between visual and textual stimulus in EEG recordings
provides valuable insights into the brain's ability to process and integrate
multi-modal information simultaneously. Inspired by this, we propose a novel
large-scale multi-modal dataset, named EIT-1M, with over 1 million
EEG-image-text pairs. Our dataset is superior in its capacity of reflecting
brain activities in simultaneously processing multi-modal information. To
achieve this, we collected data pairs while participants viewed alternating
sequences of visual-textual stimuli from 60K natural images and
category-specific texts. Common semantic categories are also included to elicit
better reactions from participants' brains. Meanwhile, response-based stimulus
timing and repetition across blocks and sessions are included to ensure data
diversity. To verify the effectiveness of EIT-1M, we provide an in-depth
analysis of EEG data captured from multi-modal stimuli across different
categories and participants, along with data quality scores for transparency.
We demonstrate its validity on two tasks: 1) EEG recognition from visual or
textual stimuli or both and 2) EEG-to-visual generation.
