HViLM-base: A Foundation Model for Viral Genomics
[!IMPORTANT]
HVUE v2 supersedes the original HVUE benchmark.
The original HVUE v1 benchmark contained substantial cross-split sequence similarity that could inflate held-out performance estimates. HVUE v2 was rebuilt using source-sequence clustering
before train/validation/test assignment and chunking, followed by exact- and near-match leakage auditing.
Use duttaprat/HVUE-v2 for current HViLM evaluation.
Model Description
HViLM (Human Virome Language Model) is a genomic foundation model adapted to virus sequences through continued pre-training of
DNABERT-2. HViLM-base was trained on approximately
5 million non-redundant virus-derived sequence fragments from the
VIRION resource, representing approximately 9,000 virus species across 45+ families.
The architecture and tokenizer remain those of DNABERT-2; continued pre-training updates the model weights using a masked-language-modeling objective on the virus-focused corpus.
Preprint: HViLM: A Foundation Model for Viral Genomics Enables Multi-Task Prediction of Pathogenicity, Transmissibility, and Host Tropism
bioRxiv 2026.03.18.712700
Authors: Pratik Dutta, Jack Vaska, Pallavi Surana, Rekha Sathian, Max Chao, Zhihan Zhou, Han Liu, and Ramana V. Davuluri
HViLM Model Family
HViLM-base is the continued-pretrained foundation model. Official task-specific models fine-tuned on HVUE v2 are released as standalone checkpoints:
| Resource | Purpose |
|---|
| HViLM-base | Continued-pretrained foundation model / sequence representations |
| HViLM-Patho | Pathogenicity classification |
| HViLM-R0 | Transmissibility classification |
| HViLM-Tropism | Human host-tropism classification |
| HVUE-v2 | Leakage-controlled benchmark |
The complete project is also grouped in the
HViLM: Human Virome Language Model collection on the
duttaprat Collections page.
Key Features
- Virus-focused continued pre-training: approximately 5M non-redundant fragments derived from VIRION-linked virus sequences.
- DNABERT-2 initialization: preserves the DNABERT-2 architecture and BPE tokenizer while adapting model weights to virus sequence data.
- Three official downstream models: pathogenicity, transmissibility, and host tropism.
- HVUE v2 evaluation: cluster-aware splitting before chunking, with multiple similarity stringencies and sequence lengths.
- Parameter-efficient downstream adaptation: official task models were trained with LoRA.
- Public reproducibility resources: base model, three task-specific checkpoints, HVUE v2 benchmark, and project code are released publicly.
Model Architecture and Continued Pre-training
HViLM-base is derived from DNABERT-2 (117M parameters).
| Property | Value |
|---|
| Architecture | MosaicBERT / DNABERT-2 |
| Parameters | ~117M |
| Hidden size | 768 |
| Transformer layers | 12 |
| Attention heads | 12 |
| Tokenization | Byte Pair Encoding (BPE) |
| Positional method | ALiBi |
| Continued-pretraining objective | Masked Language Modeling |
| Pretraining fragment length | 1000 nt |
| Final virus-focused corpus | ~5M non-redundant fragments |
| Redundancy reduction | MMseqs2 clustering at 80% identity / 80% coverage |
| Optimizer | AdamW |
| Learning rate | 5e-5 |
| Training | 10 epochs |
| Hardware | 4 × NVIDIA A100 GPUs |
| Approximate training time | 72 hours |
| Held-out MLM accuracy | 94.2% |
Sequence-length note: HViLM uses BPE tokenization, so nucleotide length and model-token length are not equivalent. The continued-pretraining corpus used 1000-nt sequence fragments; downstream configurations are described by nucleotide length in HVUE v2.
Quick Start
Extract sequence representations from HViLM-base
1import torch
2from transformers import AutoTokenizer, AutoModel
3
4model_id = "duttaprat/HViLM-base"
5
6tokenizer = AutoTokenizer.from_pretrained(
7 model_id,
8 trust_remote_code=True,
9)
10
11model = AutoModel.from_pretrained(
12 model_id,
13 trust_remote_code=True,
14)
15
16sequence = "ATGCGTACGTTAGCCGATCGATTACGCGTACGTAGCTAGCTAGCT"
17
18inputs = tokenizer(
19 sequence,
20 return_tensors="pt",
21 truncation=True,
22 padding=True,
23)
24
25with torch.no_grad():
26 outputs = model(**inputs)
27
28token_embeddings = outputs.last_hidden_state
29print(token_embeddings.shape)
For sequence-level representations, pooling strategy should be chosen according to the downstream task rather than treated as a fixed property of HViLM-base.
Use the Official Fine-tuned Models
If the goal is one of the three HVUE v2 tasks, users can load the corresponding task model directly; HViLM-base does not need to be loaded separately.
Pathogenicity
1from transformers import AutoTokenizer, AutoModelForSequenceClassification
2
3model_id = "duttaprat/HViLM-Patho"
4
5tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
6model = AutoModelForSequenceClassification.from_pretrained(
7 model_id,
8 trust_remote_code=True,
9)
Labels:
0 → NON_PATHOGENIC1 → PATHOGENIC
Transmissibility
Use:
Labels:
Host Tropism
Use:
Labels:
0 → NON_HUMAN_TROPIC1 → HUMAN_TROPIC
See the individual model cards for full usage examples and task-specific limitations.
HVUE v2 Benchmark
HVUE v2 is the current evaluation benchmark for HViLM. It replaces HVUE v1.
The benchmark was reconstructed to reduce supervised train-test leakage using the following ordering:
- consolidate and deduplicate source sequences;
- cluster source sequences with MMseqs2;
- assign complete clusters to train/validation/test splits;
- chunk sequences only after split assignment;
- remove exact duplicate chunks;
- audit cross-split exact and near matches.
HVUE v2 includes:
- Pathogenicity
- Transmissibility
- Host Tropism
Across the benchmark, configurations evaluate different sequence lengths (500, 1000, and 2000 nt where applicable), sequence-similarity stringencies, and temporal generalization where reliable collection-date metadata are available.
Primary HViLM Results
The primary results below use the standard 1000-nt configuration for each task.
| Task | HVUE v2 configuration | Accuracy | F1 | MCC | Official model |
|---|
| Pathogenicity | standard_capped_1000bp | 92.39 | 91.32 | 83.10 | HViLM-Patho |
| Transmissibility | standard_capped_1000bp | 87.50 | 86.16 | 72.66 | HViLM-R0 |
| Host Tropism | standard_95_1000bp | 96.49 | 74.49 | 48.99 | HViLM-Tropism |
The directory/configuration identifiers retain bp for release stability; manuscript and descriptive text use nt for nucleotide sequence length.
Interpretation of the HVUE v2 Results
Under leakage-controlled evaluation, the effect of virus-focused continued pre-training is task dependent:
- Pathogenicity: HViLM improves F1 by 1.28 points over vanilla DNABERT-2 (91.32 vs. 90.04).
- Transmissibility: HViLM and DNABERT-2 are close (86.16 vs. 85.81 F1), and HViLM is essentially tied with DNABERT-MB (86.16 vs. 86.15 F1).
- Host Tropism: HViLM shows the largest F1 improvement, reaching 74.49 compared with 64.82 for class-balanced DNABERT-2.
These results support a more specific conclusion than the original HVUE v1 evaluation: virus-focused continued pre-training provides its clearest benefit on the more challenging Host Tropism task, while gains on Pathogenicity and Transmissibility are smaller.
For complete baseline comparisons, hard-split evaluations, temporal evaluations, and sequence-length analyses, see the
HViLM GitHub repository and
HVUE-v2.
Training Data
Continued-pretraining corpus
HViLM-base was trained using virus sequences associated with the VIRION resource.
Processing included:
- retrieval and quality control of VIRION-linked nucleotide sequences;
- removal of short sequences and exact duplicates;
- segmentation into non-overlapping 1000-nt fragments;
- MMseqs2 clustering at 80% sequence identity and 80% coverage;
- selection of approximately 5M representative fragments for continued pre-training.
The corpus spans approximately 9,000 virus species and 45+ virus families across the Baltimore classification groups.
Interpretability
Attention-guided analyses associated with the HViLM study identified candidate sequence motifs in pathogenic coronavirus sequences, including motifs with similarity to vertebrate transcription-factor binding motifs such as IRF1, FOXQ1, and ZNF354A.
These observations are hypothesis-generating. Sequence similarity between virus motifs and host transcription-factor binding motifs does not by itself establish molecular mimicry, causal regulation, immune evasion, or another biological mechanism. Experimental validation and additional controls are required for mechanistic interpretation.
Limitations
- HVUE v2 controls supervised split leakage through source-level clustering and auditing, but sequence-similarity thresholds cannot eliminate every form of biological relatedness.
- The complete historical training exposure of the original DNABERT-2 model cannot be reconstructed; therefore, absence of all possible ancestral pretraining exposure to benchmark-related sequences cannot be guaranteed.
- Host association is biologically context-dependent and may include multi-host, zoonotic, and reverse-zoonotic relationships; the benchmark uses a simplified binary formulation.
- R₀-based transmissibility labels simplify a continuous, context-dependent epidemiological quantity into a binary benchmark task.
- Performance differences between closely matched models should not be interpreted as statistically meaningful without uncertainty estimates or repeated evaluations.
- Attention-based motif analyses should be considered exploratory rather than direct evidence of mechanism.
- HViLM predictions are research outputs and are not intended to replace experimental, clinical, epidemiological, or public-health assessment.
Reproducibility and Resources
Citation
If you use HViLM in your research, please cite:
1@article{dutta2026hvilm,
2 title={HViLM: A foundation model for viral genomics enables multi-task prediction of pathogenicity, transmissibility, and host tropism},
3 author={Dutta, Pratik and Vaska, Jack and Surana, Pallavi and Sathian, Rekha and Chao, Max and Zhou, Zhihan and Liu, Han and Davuluri, Ramana V},
4 journal={bioRxiv},
5 pages={2026--03},
6 year={2026},
7 publisher={Cold Spring Harbor Laboratory}
8}
If you use DNABERT-2 directly or build on its architecture, please also cite the DNABERT-2 publication.
Model Card Authors
- Pratik Dutta — Stony Brook University
- Ramana V. Davuluri — Stony Brook University
Contact
Acknowledgments
HViLM builds on DNABERT-2 by Zhou et al. Continued-pretraining data were derived from the VIRION resource maintained by the Viral Emergence Research Initiative (Verena).
License
HViLM-base is released under the Apache License 2.0.
Disclaimer
HViLM is a research model for computational biology. It should not be used as the sole basis for clinical, diagnostic, epidemiological, biosurveillance, or public-health decisions. Model outputs should be interpreted alongside appropriate biological evidence and expert assessment.
