JanusDNA: A Powerful Bi-directional Hybrid DNA Foundation Model
Journal:
arXiv
Published Date:
May 22, 2025
Abstract
Large language models (LLMs) have revolutionized natural language processing
and are increasingly applied to other sequential data types, including genetic
sequences. However, adapting LLMs to genomics presents significant challenges.
Capturing complex genomic interactions requires modeling long-range
dependencies within DNA sequences, where interactions often span over 10,000
base pairs, even within a single gene, posing substantial computational burdens
under conventional model architectures and training paradigms. Moreover,
standard LLM training approaches are suboptimal for DNA: autoregressive
training, while efficient, supports only unidirectional understanding. However,
DNA is inherently bidirectional, e.g., bidirectional promoters regulate
transcription in both directions and account for nearly 11% of human gene
expression. Masked language models (MLMs) allow bidirectional understanding but
are inefficient, as only masked tokens contribute to the loss per step. To
address these limitations, we introduce JanusDNA, the first bidirectional DNA
foundation model built upon a novel pretraining paradigm that combines the
optimization efficiency of autoregressive modeling with the bidirectional
comprehension of masked modeling. JanusDNA adopts a hybrid Mamba, Attention and
Mixture of Experts (MoE) architecture, combining long-range modeling of
Attention with efficient sequential learning of Mamba. MoE layers further scale
model capacity via sparse activation while keeping computational cost low.
Notably, JanusDNA processes up to 1 million base pairs at single nucleotide
resolution on a single 80GB GPU. Extensive experiments and ablations show
JanusDNA achieves new SOTA results on three genomic representation benchmarks,
outperforming models with 250x more activated parameters. Code:
https://github.com/Qihao-Duan/JanusDNA
