Learn to Enhance Sparse Spike Streams.

Journal: IEEE transactions on pattern analysis and machine intelligence
Published Date: May 1, 2026

Abstract

High-speed vision tasks have long been a challenge in computer vision. Recently, the spike camera has shown great potential in these tasks due to its high temporal resolution. Unlike traditional cameras, it emits asynchronous spike signals to capture visual information. However, under low-light conditions, spike signals become highly sparse, and the sparse spike stream severely hinders the effectiveness of existing spike-based methods in high-speed scenarios. To address this challenge, we introduce SS2DS, the first deep learning framework that enhances sparse spike streams into dense spike streams. SS2DS first estimates the spike firing frequency within sparse streams. Subsequently, the spike firing frequency is enhanced by a neural network. Finally, SS2DS decodes the enhanced spike stream from the enhanced spike firing frequency sequence. SS2DS can adjust the temporal distribution of sparse spike streams and improve the performance degradation of existing methods in low-light and high-speed scenarios. To evaluate sparse spike stream enhancement, we construct both synthetic and real sparse spike stream datasets. By comparing the reconstruction results, enhanced spike streams achieve an average improvement of +0.78 MA, -18.42 BRISQUE, and -1.42 NIQE over sparse spike streams. Moreover, the enhanced spike streams also benefit other spike-based vision tasks, such as 3D reconstruction (+1.325 dB PSNR, +0.005 SSIM, and -0.01 LPIPS) and superresolution (+0.63 MA, -13.67 BRISQUE, and -1.28 NIQE).

Authors

  • Liwen Hu
  • Yijia Guo
  • Mianzhi Liu
    National Biomedical Imaging Center, State Key Laboratory of Membrane Biology, Institute of Molecular Medicine, Peking-Tsinghua Center for Life Sciences, College of Future Technology, Peking University, Beijing, China.
  • Yiming Fan
    Dalian Harmony Medical Testing Laboratory Co., Ltd, 116620 Dalian City, Liaoning Province, China.
  • Rui Ma
    Institute of Agricultural Resources and Regional Planning, Chinese Academy of Agricultural Sciences, Beijing, China.
  • Shengbo Chen
    School of Computer and Information Engineering, Henan University, Kaifeng 475001, China.
  • Lei Ma
    School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu 611731, Sichuan, China. Electronic address: [email protected].
  • Tiejun Huang
    National Engineering Laboratory for Video Technology, School of Electronics Engineering and Computer Science, Peking University, Beijing, China; Peng Cheng Laboratory, Shenzhen, China.

Keywords

No keywords available for this article.

External Resources

Popular Topics
Recent Journals