CoQMoE: Co-Designed Quantization and Computation Orchestration for Mixture-of-Experts Vision Transformer on FPGA
Journal:
arXiv
Published Date:
Jun 10, 2025
Abstract
Vision Transformers (ViTs) exhibit superior performance in computer vision
tasks but face deployment challenges on resource-constrained devices due to
high computational/memory demands. While Mixture-of-Experts Vision Transformers
(MoE-ViTs) mitigate this through a scalable architecture with sub-linear
computational growth, their hardware implementation on FPGAs remains
constrained by resource limitations. This paper proposes a novel accelerator
for efficiently implementing quantized MoE models on FPGAs through two key
innovations: (1) A dual-stage quantization scheme combining
precision-preserving complex quantizers with hardware-friendly simplified
quantizers via scale reparameterization, with only 0.28 $\%$ accuracy loss
compared to full precision; (2) A resource-aware accelerator architecture
featuring latency-optimized streaming attention kernels and reusable linear
operators, effectively balancing performance and resource consumption.
Experimental results demonstrate that our accelerator achieves nearly 155
frames per second, a 5.35$\times$ improvement in throughput, and over $80\%$
energy reduction compared to state-of-the-art (SOTA) FPGA MoE accelerators,
while maintaining $<1\%$ accuracy loss across vision benchmarks. Our
implementation is available at https://github.com/DJ000011/CoQMoE.
