Views
No views yet
Initialization: pretrained from Nucleotide Transformer, then fine-tuned with masked diffusion objective on mammalian DNA.
| Parameters | Hidden | Layers | Heads | Max Length | Vocab |
|---|---|---|---|---|---|
| ~50M | 512 | 12 | 16 | 2,048 | 4,107 |
pip install transformers torch tqdm1import sys
2import torch
3from transformers import AutoTokenizer, AutoModelForMaskedLM
4
5model_name = "Hengchang-Liu/D3LM-from-nt"
6model = AutoModelForMaskedLM.from_pretrained(model_name, trust_remote_code=True).eval()
7tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
8
9# Import MDMGenerationConfig from the model's auto-downloaded module
10MDMGenerationConfig = getattr(sys.modules[type(model).__module__], "MDMGenerationConfig")
11
12# Unconditional generation: create a fully-masked prompt of desired length
13length = 200
14input_ids = torch.full((1, length), tokenizer.mask_token_id, dtype=torch.long)
15
16config = MDMGenerationConfig(
17 mask_token_id=tokenizer.mask_token_id,
18 max_length=length,
19 steps=50,
20 temperature=1.0,
21 top_p=0.9,
22 alg="random",
23 num_return_sequences=4,
24 return_dict_in_generate=True,
25)
26
27with torch.no_grad():
28 outputs = model.diffusion_generate(inputs=input_ids, generation_config=config)
29
30for i, seq in enumerate(outputs.sequences):
31 print(f">{i}
32{tokenizer.decode(seq, skip_special_tokens=True).replace(' ', '')}")| Parameter | Default | Description |
|---|---|---|
steps | 50 | Diffusion denoising steps |
temperature | 1.0 | Sampling temperature |
top_p | 0.9 | Nucleus sampling cutoff |
top_k | 0 | Top-k cutoff (0 = off) |
alg | "random" | Unmasking order: random, entropy, maskgit_plus, topk_margin, origin, p2 |
alg_temp | 0.9 | Gumbel temperature for confidence ordering (0 = deterministic) |
@misc{yang2026d3lmdiscretednadiffusion,
title={D3LM: A Discrete DNA Diffusion Language Model for Bidirectional DNA Understanding and Generation},
author={Zhao Yang and Hengchang Liu and Chuan Cao and Bing Su},
year={2026},
eprint={2603.01780},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2603.01780},
}