A CNN-transformer hybrid approach for decoding visual neural activity into text.

Journal: Computer methods and programs in biomedicine
Published Date: Dec 14, 2021

Abstract

BACKGROUND AND OBJECTIVE: Most studies used neural activities evoked by linguistic stimuli such as phrases or sentences to decode the language structure. However, compared to linguistic stimuli, it is more common for the human brain to perceive the outside world through non-linguistic stimuli such as natural images, so only relying on linguistic stimuli cannot fully understand the information perceived by the human brain. To address this, an end-to-end mapping model between visual neural activities evoked by non-linguistic stimuli and visual contents is demanded.

Authors

  • Jiang Zhang
    College of Electrical Engineering and Information Technology, Sichuan University, Chengdu, 610065, China.
  • Chen Li
    School of Computer Science and Technology, Xi'an Jiaotong University, Xi'an, Shaanxi, China.
  • Ganwanming Liu
    College of Electrical Engineering, Sichuan University, Chengdu 610065, China.
  • Min Min
    College of Electrical Engineering, Sichuan University, Chengdu 610065, China.
  • Chong Wang
    Shandong Xinhua Pharmaceutical Co., Ltd., No. 1, Lu Tai Road, High Tech Zone, Zibo 255199, China.
  • Jiyi Li
    MOE Key Lab for Neuroinformation, University of Electronic Science and Technology of China, Chengdu, 610054, People's Republic of China.
  • Yuting Wang
    Respiratory Department, Dongzhimen Hospital Affiliated to BUCM, Beijing, China.
  • Hongmei Yan
    Ministry of Education Key Laboratory of Metabolism and Molecular Medicine, Department of Endocrinology and Metabolism, Zhongshan Hospital, Fudan University, Shanghai, China.
  • Zhentao Zuo
    State Key Laboratory of Brain and Cognitive Science, Beijing MR Center for Brain Research, Institute of Biophysics, Chinese Academy of Sciences, Beijing, 100101, China. ztzuo@bcslab.ibp.ac.cn.
  • Wei Huang
    Shaanxi Institute of Flexible Electronics, Northwestern Polytechnical University, 710072 Xi'an, China.
  • Huafu Chen
    Key laboratory for NeuroInformation of Ministry of Education, School of Life Science and Technology and Center for Information in BioMedicine, University of Electronic Science and Technology of China, Chengdu 610054, PR China. Electronic address: chenhf@uestc.edu.cn.

Keywords

External Resources

Popular Topics
Recent Journals