mlm-baseline-hg38
A masked nucleotide language model for the Homo sapiens genome (hg38).
What it predicts
The model is a masked language model over DNA: it learns the distribution of each nucleotide given its surrounding sequence context. At every position it outputs a probability distribution over the four nucleotides (A, C, G, T) — what base it expects there from the context alone. It is trained on the Homo sapiens genome, and is useful for scoring how unexpected a variant is (its likelihood under the model) and as a sequence feature extractor.
Inputs and outputs
The model takes a LongTensor of shape (batch, length) that encodes DNA over a six-token alphabet: A, C, G, T, N, and a padding/mask symbol, mapped to indices 0 through 5. It returns a FloatTensor of shape (batch, length - 1548, 4): at each in-bounds position, four logits over [A, C, G, T]. A softmax over them is the predicted nucleotide distribution. Because it is a masked model, to read the prediction for a position you first replace that position's input token with the mask token (index 5).
Status
This revision is a partial run: it was released at step 93,750, roughly half of the configured cosine cycle, and the learning rate had not annealed. It is published so downstream benchmarks can run; a completed checkpoint is expected to replace it in a later revision. Pin a revision if you need reproducibility.
Training
Trained by masked language modeling: about 15% of input tokens are replaced with the mask token (index 5), and the loss is the cross-entropy between the model's per-position distribution and the true nucleotide at the masked positions. The network is reverse-complement equivariant by construction, so a sequence and its reverse complement receive consistent predictions.
How to use
1from huggingface_hub import hf_hub_download
2from safetensors.torch import load_file
3import importlib.util, torch
4
5repo_id = "songlab/mlm-baseline-hg38"
6model_path = hf_hub_download(repo_id=repo_id, filename="model.py")
7weights_path = hf_hub_download(repo_id=repo_id, filename="model.safetensors")
8spec = importlib.util.spec_from_file_location("released_model", model_path)
9module = importlib.util.module_from_spec(spec); spec.loader.exec_module(module)
10
11model = module.CNN() # defaults match the released checkpoint
12model.load_state_dict(load_file(weights_path))
13model.eval()
14
15# Input: a (batch, length) LongTensor over [A, C, G, T, N, pad], length >= 1549.
16x = torch.randint(0, 4, (1, 2000))
17with torch.no_grad():
18 output = model(x) # (1, length - 1548, 4): per-position logits
19 embeddings = model.encode(x) # (1, length - 1548, 512): per-position features
This is a masked language model: to read the prediction for a position, replace that position's input token with the mask token (index 5) before the forward pass, then take a softmax over the four output logits. The output window is centered, so input position p maps to output index p - 774:
1masked = x.clone()
2masked[0, 874] = 5 # mask one position
3with torch.no_grad():
4 probs = model(masked).softmax(-1)
5 prediction = probs[0, 100] # distribution over [A, C, G, T] at the masked position
Outputs cover only positions whose full 1549 bp context is in bounds. encode returns the per-position features (use it for embeddings).
Files
model.py: a self-contained CNN class with no dependencies beyond PyTorch.model.safetensors: the released weights (~1.06 GB).config.json: architecture hyperparameters, tokenization, and training metadata.
License
MIT
