Accurate INT8 Training Through Dynamic Block-Level Fallback
Journal:
arXiv
Published Date:
Mar 11, 2025
Abstract
Transformer models have achieved remarkable success across various AI
applications but face significant training costs. Low-bit training, such as
INT8 training, can leverage computational units with higher throughput, and has
already demonstrated its effectiveness on GPT2 models with block-level
quantization. However, it struggles with modern Transformer variants
incorporating GLU units. This is because those variants demonstrate complex
distributions of activation outliers. To address the challenge, we propose
Fallback Quantization, implementing mixed-precision GEMM that dynamically falls
back 8-bit to 16-bit for activation blocks containing outliers. Experiments
show that our approach is robustly competent in both fine-tuning and
pretraining settings. Moreover, our method achieves a 1.57x end-to-end training
speedup on RTX4090 GPUs.
