Fast and Low-Cost Genomic Foundation Models via Outlier Removal
Journal:
arXiv
Published Date:
May 1, 2025
Abstract
To address the challenge of scarce computational resources in genomic
modeling, we introduce GERM, a genomic foundation model with strong compression
performance and fast adaptability. GERM improves upon models like DNABERT-2 by
eliminating outliers that hinder low-rank adaptation and post-training
quantization, enhancing both efficiency and robustness. We replace the vanilla
attention layer with an outlier-free mechanism inspired by associative memory
models. By removing outliers during both pre-training and fine-tuning, this
approach accelerates adaptation, reduces computational costs, and enhances
quantization robustness within acceptable loss margins. Additionally, we
propose GERM-T, a strategy that employs small-step continual learning within
the outlier-free framework, leveraging original checkpoints to avoid retraining
from scratch. Empirically, GERM improves fine-tuning performance by 37.98% and
quantization by 64.34% over the baseline model. It also reduces average
kurtosis by 92.14% and maximum infinity norm by 82.77%. Compared to leading
methods, GERM consistently delivers superior performance, offering a practical
solution for genomic modeling in resource-constrained settings. Code is
available at https://github.com/MAGICS-LAB/GERM.
