Model Overview
JEPA-DNA-DNABERT2 is the DNABERT-2 backbone of the JEPA-DNA family, a model-agnostic continual pre-training framework that grounds genomic foundation models by coupling standard token-level DNA language modeling with a Joint-Embedding Predictive Architecture (JEPA). Where conventional genomic models rely purely on masked language modeling and prioritize local token reconstruction, JEPA-DNA additionally supervises the model's global sequence embedding in a latent space, forcing it to predict the functional representation of masked genomic segments rather than their literal tokens. This shifts the learning signal from token recovery toward semantic alignment, yielding more linearly-separable and biologically relevant features while preserving the generative precision of the underlying backbone. This checkpoint is an exact parameter match to the original research work described in the
JEPA-DNA paper, and the code for using it can be found in the
official JEPA-DNA Github repository.
Description:
JEPA-DNA-DNABERT2 is the DNABERT-2 variant of JEPA-DNA, a genomic foundation model training framework that adds a Joint-Embedding Predictive Architecture (JEPA) objective to standard DNA language-model pretraining. It applies a continual pre-training stage on top of the DNABERT-2 backbone — a 117M-parameter Transformer encoder trained with a masked language modeling (MLM) objective and BPE tokenization — combining token-level learning with latent-space prediction to improve functional genomic representations.
This model is ready for non-commercial use.
License/Terms of Use:
Governing Terms: Use of this model is governed by the
NVIDIA non-commercial license, subject to completion of NVIDIA's applicable internal legal and open source release review process.
Deployment Geography:
Global
Use Case:
JEPA-DNA checkpoints are intended for genomics research, representation learning, and downstream sequence modeling workflows. Typical uses include feature extraction, linear probing, continual pre-training studies, and zero-shot embedding-based scoring of sequence perturbations. This model is not intended to be used as diagnostic tools, clinical decision systems, or validated medical products.
Release Date:
References(s):
Model Architecture:
JEPA-DNA-DNABERT2 uses DNABERT-2, a 117M-parameter Transformer encoder trained with a masked language modeling (MLM) objective and BPE tokenization, with sequence aggregation based on the [CLS] token embedding. JEPA-DNA augments this backbone with a target encoder (an exponential-moving-average copy of the backbone) and a lightweight 4-layer Transformer predictor that reconstructs the target's latent representations in a reduced embedding space. Training uses span-based masking and re-masking to prevent trivial recovery of masked targets, and proceeds in two stages: first optimizing the predictor with a frozen backbone, then joint optimization of predictor and backbone under a composite objective combining the native MLM loss, a JEPA cosine-similarity latent-prediction loss, and VICReg-style variance and covariance regularization:
$$\min_{\theta,\phi}; \mathcal{L}{total} = \lambda_1 \mathcal{L}{llm} + \lambda_2 \mathcal{L}{jepa} + \lambda_3 \mathcal{L}{var} + \lambda_4 \mathcal{L}_{cov}$$
| Model Name | Parameters |
|---|
| JEPA-DNA-DNABERT2 | 1.17 × 10⁸ |
Input:
Input Type(s): Text (genomic DNA sequences, tokenized)
Input Format: DNA String
Input Parameters: 1D
Other Properties Related to Input: Input DNA sequences follow the DNABERT-2 backbone's context length, BPE tokenization, and preprocessing, with sequence aggregation based on the [CLS] token embedding.
Output:
Output Type(s): Embeddings / latent sequence representations
Output Format: Embeddings (2D, sequence-length × hidden-dim) and sequence representations (1D, hidden-dim)
Output Parameters: 2D (embeddings), 1D (sequence representation)
Other Properties Related to Output: Outputs are sequence-level embeddings and hidden states that can be used for linear probing, embedding-based scoring of sequence perturbations, and retrieval-style tasks; they do not directly produce labels but are intended for downstream models or evaluators.
Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated systems. By leveraging NVIDIA's hardware (e.g. GPU cores) and software frameworks (e.g., CUDA libraries), the model achieves faster training and inference times compared to CPU-only solutions.
Software Integration:
Runtime Engine(s):
Supported Hardware Microarchitecture Compatibility:
- NVIDIA Blackwell
- NVIDIA A10G
Preferred/Supported Operating System(s):
The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment. Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.
Model Version(s):
Training, Testing, and Evaluation Datasets:
Training Dataset:
Properties: Continual pre-training uses a subset of the DNABERT-2 multi-species corpus (human and five model organisms), approximately 4.76M sequences, following the original DNABERT-2 pretraining data. Content is genomic DNA sequences encoded as text-like sequences of nucleotide tokens. By maintaining data parity with the original baseline, the continual-training stage isolates the impact of the JEPA-DNA objective.
Data Modality
- Text (genomic DNA sequences)
Training Data Size: Approximately 4.76M DNA sequences.
Data Collection Method by dataset:
- Hybrid: Automated, Automatic/Sensors
Labeling Method by dataset:
Evaluation Datasets:
Evaluation is performed across 17 genomic benchmark tasks provided by
GFMBench-API, covering supervised linear probing (9 tasks) and zero-shot (8 tasks) protocols.
Benchmark Score: For the DNABERT-2 backbone, JEPA-DNA yields a mean AUROC gain of about 2.6% across evaluated tasks (with AUROC gains of up to +8.0% on individual tasks) and a mean zero-shot AUROC gain of about 4.1%. Metrics reported are AUROC, AUPRC, and Matthews Correlation Coefficient (MCC).
Data Collection Method by dataset:
- Hybrid: Automated, Automatic/Sensors
Labeling Method by dataset:
Inference:
Acceleration Engine: PyTorch - 2.6.0
Test Hardware: NVIDIA Blackwell (B300 SXM6 GPU)
Ethical Considerations:
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
Users are responsible for ensuring the physical properties of model-generated molecules are appropriately evaluated and comply with applicable safety regulations and ethical standards.
For more detailed information on ethical considerations for this model, please see the Model Card++ Bias, Explainability, Safety & Security, and Privacy Subcards.
Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns
here.
