You can deploy this model and start earning money today!

yagizdevre/
nanoprot-gpt2-S

Name: nanoprot-gpt2-S
Availability: InStock

nanoprot

protein-language-model

Views

No views yet

nanoprot-gpt2-S

A protein language model (gpt2 architecture) — 36.1M (36,090,440) parameters, trained on 433.1M (433,085,280) UniRef50 residues.

nanoprot-gpt2-S is part of the nanoprot suite: a Pythia-style matrix of protein language models spanning three architectures (gpt2, esm2, mamba) and four scales (XS/S/M/L), each trained from scratch on UniRef50 under a matched, Chinchilla-style data budget. The suite is built for controlled comparison — same data, same tokenizer, one variable at a time.

Headline result

Validation bits-per-residue (lower is better): 3.8807 ± 0.0002 (n=3 seeds)

Directly comparable to other autoregressive nanoprot models (gpt2, mamba) — same 33-token vocabulary, same AR objective, same data budget. Not comparable to the esm2 (masked-LM) models, whose metric is a different quantity.

Model details


Architecture	`gpt2`
Objective	AR (causal-language-modeling)
Scale rung	S
Parameters	36.1M (36,090,440)
Layers (depth)	9
Hidden size (d_model)	576
Attention heads	9
Max sequence length	512
Vocabulary	33-token residue alphabet (ESM-2)
MLP activation	relu_squared
logit softcap	15.0
window pattern	L
Precision	bf16

Training


Data	UniRef50 release 2026_01 (28-Jan-2026), 60,251,814 sequences; held-out final shard for validation
Tokenizer	`esm2` — 33-token residue alphabet (shared across the whole suite)
Optimizer	Muon (matrices) + AdamW (embeddings/scalars), weight_decay=0.1
Batch size	524,288 residues/step
Optimizer steps	826
Residues seen	433.1M (433,085,280)
Param/data ratio	12.0 (Chinchilla-style)
Total FLOPs	1.070e+17
Wall-clock	0.09 h (6 min) on 4 GPU(s)
Seed	0 (siblings: see below)
nanoprot version	0.5.0

Evaluation

Evaluated on a held-out UniRef50 shard. Validation bits-per-residue (lower is better): 3.8807 ± 0.0002 (n=3 seeds).

Directly comparable to other autoregressive nanoprot models (gpt2, mamba) — same 33-token vocabulary, same AR objective, same data budget. Not comparable to the esm2 (masked-LM) models, whose metric is a different quantity.

Intended use & limitations

Research use: learning protein representations, extracting residual-stream features, mechanistic-interpretability probing, and architecture comparison. Trained only on UniRef50 sequences — not for clinical or diagnostic use, and not aligned to any downstream task out of the box.

How to load

Install nanoprot (pip install nanoprot), download this repo, and point the arch-aware loader at the folder — it works for any nanoprot architecture (gpt2 / esm2 / mamba), reading the embedded config and selecting the right tokenizer automatically.

python

from nanoprot.training.checkpoint import load_pretrained

model, cfg, meta, tokenizer = load_pretrained(
    "path/to/this/repo", device="cpu", return_tokenizer=True,
)
model.eval()
# meta carries the trained-artifact facts (params, FLOPs, val metric, ...)

The nanoprot suite

Hub: yagizdevre/nanoprot-gpt2-S (seed 0 is the default; siblings on branches seed1, seed2). This model's sibling seeds:

nanoprot-gpt2-S-s0 — final bits-per-residue 3.8808
nanoprot-gpt2-S-s1 — final bits-per-residue 3.8805
nanoprot-gpt2-S-s2 — final bits-per-residue 3.8807

The full suite spans {gpt2, esm2, mamba} x {XS, S, M, L} x {seed 0,1,2}. See the nanoprot repository for the complete grid and the scaling-curve comparisons.

Citation

bibtex

@software{nanoprot,
  author  = {Devre, H. Yagiz},
  title   = {nanoprot: a minimal training framework for protein language models},
  year    = {2026},
  url     = {https://github.com/ygzdvr/nanoprot}
}

Reproducibility

Trained with nanoprot v0.5.0 (corpus prepared 2026-06-01T12:51:52+00:00). The exact, complete training config is in config.yaml (also embedded in meta_000826.json). Re-train with:

python -m scripts.train --config config.yaml