Minimal HuggingFace port of the 1.5-7B-8K variant of Evo 1.5 -- the ~7B-parameter Evo 1 model continued on approximately 50% more pretraining data at an 8,192-token context.

Why this port?

evo-design/evo-1.5-8k-base ships a trust_remote_code HF implementation but it has four gaps that force every downstream user to monkey-patch the model:

output_hidden_states=True is hardcoded to None (intermediate embeddings require forward hooks).
output_attentions=True is unsupported (flash-attn discards the (B, H, T, T) matrix; users must patch the attention module).
attn_implementation cannot be switched at load time - flash_attn is mandatory at every attention layer.
The bare backbone is not exposed via AutoModel.from_pretrained; only the LM-head wrapper exists.

This repo fixes all four. The math is bit-exact with the evo-design reference (max_abs_diff = 0.000e+00 at every layer; see Parity Verification).

Architecture

Parameter	Value
Total parameters	~7B
Layers	32
Attention heads	32
Embedding dimension	4096
FFN hidden dimension	10 928 (gated GELU)
Vocabulary size	512 (UTF-8 byte-level)
Attention layer indices	[8, 16, 24]
Hyena layer indices	all others
Hyena state size	8
Positional encoding	RoPE (base = 10000)
Normalization	RMSNorm (eps = 1e-6)
Architecture	StripedHyena (29 Hyena blocks + 3 causal MHA blocks)
Max sequence length	8 192 (training context)
Training dtype	bfloat16 (Hyena modal-form `poles` / `residues` kept in fp32)

Vocabulary: raw UTF-8 byte IDs 0-255; the embedding/output vocabulary is padded to 512 entries. Byte 0 is EOD/EOS and byte 1 is PAD.

Architecturally identical to Evo1-1-7B-8K; only the trained weights differ (Evo 1.5 = Evo 1 (8k) + ~50% more pretraining tokens).

Pretraining

Objective: causal byte-level next-token prediction.
Data: OpenGenome, the prokaryotic whole-genome dataset assembled for Evo 1 (~80 000 prokaryotic genomes plus ~2 million phage and plasmid sequences), with training extended by an additional 50% - totaling ~450 billion tokens (vs. ~300 B for Evo 1).
Initialization: continued pretraining from evo-1-8k-base at 8 192-token context (i.e. Evo 1.5 is not trained from scratch; the additional ~150 B tokens are appended to Evo 1's 300 B-token training run).
Source checkpoint: evo-design/evo-1.5-8k-base@main.

Downstream use

Evo 1.5 is the model used in Merchant et al., 2025 (Nature) for semantic mining of functional de novo genes - generating new candidate genes conditioned on genomic context. It was used to produce SynGenome, the first large-scale AI-generated genomic database (over 120 B base pairs of synthetic DNA, organised by predicted function, protein domain, and host species).

Parity Verification

Hidden-state representations verified bit-exact (max_abs_diff = 0.000e+00) to the evo-design reference at all 34 representation levels (token embedding + each of the 32 hybrid blocks + final RMSNorm), using attn_implementation="flash_attention_2" in bf16 (matches the reference's backend choice and the trained dtype). Logits from Evo1ForCausalLM were also verified bit-exact (top-1 agreement: 128/128 positions). Verified on H100 with PyTorch 2.7.1 / CUDA 12.9.

Numerical equivalence across attention backends

flash_attention_2 is bit-exact with the original togethercomputer / evo-design implementations (same CUDA kernel). The sdpa and eager backends use different kernels (PyTorch's bundled flash kernel and pure-PyTorch matmul, respectively); these compute mathematically equivalent attention but accumulate floating-point operations in slightly different orders, producing per-block diffs at the bf16 noise floor (relative error roughly 1e-4 to 1e-2).

Unlike a standard transformer, where attention is softmax-bounded and per-block diffs stay small through the stack, StripedHyena's Hyena layers use an unbounded-gain IIR filter (no softmax) - so any small per-attention-block diff gets amplified by Hyena's filter gain. Across 32 layers this compounds to ~1% relative error in the intermediate residual stream, though the final post-RMSNorm output is bounded. Use flash_attention_2 if you need to match the reference's activations bit-for-bit.

Related Models

See the full Evo1 collection on the Hub.

Model	Context	Notes
Taykhoom/Evo1-1-7B-8K	8 192	Original Evo 1 base model (8k context).
Taykhoom/Evo1-1-7B-131K	131 072	Long-context Evo 1 with linearly-scaled RoPE (131k context).
Taykhoom/Evo1-1.5-7B-8K	8 192	Evo 1.5: Evo 1 (8k) further trained on ~50% more pretraining tokens.

Usage

Note on dtype. Evo1 was trained in bfloat16, with the Hyena poles / residues (modal-form filter parameters) and rotary inv_freq kept in fp32 for numerical stability. from_pretrained defaults to bf16 and the bundled remote model code automatically restores these fp32 invariants after checkpoint loading and after later .to(...), .half(), or .bfloat16() calls; no manual repair call is needed.

Note on attention backend. By default, from_pretrained selects attn_implementation="sdpa" (PyTorch's bundled scaled-dot-product-attention) - this works out of the box without flash_attn installed. The original togethercomputer / evo-design implementations use flash_attn unconditionally; for bit-exact reproduction of reference outputs, explicitly pass attn_implementation="flash_attention_2" (and pip install flash-attn). See Numerical equivalence across attention backends for the magnitude of the difference.

Embedding generation (no LM head)

python

import torch
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("Taykhoom/Evo1-1.5-7B-8K", trust_remote_code=True)
model = AutoModel.from_pretrained(
    "Taykhoom/Evo1-1.5-7B-8K",
    trust_remote_code=True,
    attn_implementation="flash_attention_2",  # bit-exact with reference; or omit to default to "sdpa"
).cuda().eval()

seqs = ["ACGTACGTACGT", "GGGTTTAAACCC"]
inputs = tokenizer(seqs, return_tensors="pt", padding=True).to(model.device)

with torch.no_grad():
    out = model(**inputs, output_hidden_states=True)

last_hidden  = out.last_hidden_state  # (B, T, 4096)
all_layers   = out.hidden_states      # tuple of (B, T, 4096), len = 34 (embed + 32 blocks + post-norm)
layer_12_emb = all_layers[12]         # often used as a "middle" representation

LM logits

python

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Taykhoom/Evo1-1.5-7B-8K", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    "Taykhoom/Evo1-1.5-7B-8K",
    trust_remote_code=True,
    attn_implementation="flash_attention_2",
).cuda().eval()

inputs = tokenizer(["ACGT"], return_tensors="pt").to(model.device)
with torch.no_grad():
    logits = model(**inputs).logits   # (1, T, 512)

Faster attention backends

Omit attn_implementation to use PyTorch SDPA, pass "eager" for materialized attention probabilities, or pass "flash_attention_2" (with flash-attn installed) for bit-exact parity with the upstream implementation.

Generation

python

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Taykhoom/Evo1-1.5-7B-8K", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    "Taykhoom/Evo1-1.5-7B-8K",
    trust_remote_code=True,
    attn_implementation="flash_attention_2",
).cuda().eval()

inputs = tokenizer(["ACGT"], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=True, top_k=4, temperature=1.0)
print(tokenizer.decode(out[0]))

generation_config.json ships with eos_token_id = 0 (the EOD byte) and pad_token_id = 1 so model.generate() stops naturally at the trained end-of-document token without needing extra kwargs. Note that the tokenizer itself does not add an EOS at encoding time - this matches the original Evo1 inference pipeline (only generation stops on EOS; embedding/scoring uses raw byte input).

Cache limitation: cached generation is reliable for one prompt or an equal-length batch. Variable-length padded batches are not cache-safe; run those prompts separately. Start each new generation with past_key_values=None rather than reusing or resetting an Evo1Cache.

Attention weights

python

import torch
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("Taykhoom/Evo1-1.5-7B-8K", trust_remote_code=True)
model = AutoModel.from_pretrained(
    "Taykhoom/Evo1-1.5-7B-8K",
    trust_remote_code=True,
    attn_implementation="eager",  # avoids the automatic eager fallback
).cuda().eval()

inputs = tokenizer(["ACGTACGT"], return_tensors="pt").to(model.device)
with torch.no_grad():
    out = model(**inputs, output_attentions=True)

# out.attentions is a tuple of length 32. Entries at indices not in [8, 16, 24]
# are None (Hyena blocks have no attention matrix). Entries at [8, 16, 24] are
# (B, num_heads, T, T) tensors.
attn_block_8 = out.attentions[8]

Multi-GPU loading (optional)

Loading via accelerate's device_map is supported (_no_split_modules is set so each AttentionBlock / ParallelGatedConvBlock stays atomic on one device, with hidden state automatically transferred across device boundaries):

python

import torch
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("Taykhoom/Evo1-1.5-7B-8K", trust_remote_code=True)
model = AutoModel.from_pretrained(
    "Taykhoom/Evo1-1.5-7B-8K",
    trust_remote_code=True,
    attn_implementation="flash_attention_2",
    device_map="auto",     # auto-shard across all visible GPUs; falls back to single GPU if only one is present
).eval()

Requires pip install accelerate.

Fine-tuning

Standard HuggingFace conventions. For sequence-level tasks, take the final last_hidden_state (or any intermediate hidden_states[i]) and feed it into a downstream head.

Implementation Notes

HF base-model contract. base_model_prefix = "backbone" exposes the raw StripedHyena module through .base_model. Its public forward accepts standard HF arguments (input_ids, attention_mask, past_key_values, use_cache, output_hidden_states, output_attentions, return_dict) and returns BaseModelOutputWithPast; AutoModel delegates to this same path.
Custom attention module (attention.py). Replaces flash_attn.modules.mha.MHA with a small in-repo MHA class that supports attn_implementation="eager" / "sdpa" / "flash_attention_2". Learned parameter names (Wqkv, out_proj) are preserved so existing checkpoints load unchanged. When output_attentions=True, the sdpa and flash paths automatically fall back to eager so the attention matrix is materialized.
Custom rotary embedding (rotary.py). inv_freq is treated as non-persistent runtime state and reconstructed from base and dim in fp32 after checkpoint loading and whenever the RoPE cache is rebuilt. When flash_attn is installed we delegate application to its Triton kernel; the pure-PyTorch fallback likewise performs the rotary multiply in fp32 before casting back. This avoids the per-layer error caused by bf16 frequency or multiplication rounding.
Hyena engine (engine.py). Ported from the togethercomputer reference (FFT-based long convolution, modal-form prefill).
Cache subclass (cache.py). Evo1Cache(transformers.cache_utils.Cache) wraps the two block-type-specific inference param dataclasses (InferenceParams for attention KV cache, RecurrentInferenceParams for Hyena FIR window + IIR modal state). Exposes get_seq_length() / get_max_cache_shape() so HF's model.generate() can introspect cache state; falls through to cache["mha"] / cache["hyena"] for the model internals.
Tokenizer (tokenization_evo1.py). Byte-level UTF-8 with vocab_size = 512. Pad token is byte \x01. No CLS, no EOS appended at encoding time (matches original Evo1 inference). The _decode method is numpy-2.x compatible (the original np.uint8.clip(min=32, max=512) was an overflow on numpy 2).
Dependencies. torch, transformers, numpy, safetensors, huggingface_hub (only for from_pretrained downloads). flash_attn is only required if you pass attn_implementation="flash_attention_2".

Citation

bibtex

@article{merchant2025_evo_1_5,
  title   = {Semantic design of functional de novo genes from a genomic language model},
  author  = {Merchant, Aditi T. and King, Samuel H. and Nguyen, Eric and Hie, Brian L.},
  journal = {Nature},
  volume  = {649},
  number  = {8097},
  pages   = {749--758},
  year    = {2025},
  doi     = {10.1038/s41586-025-09749-7}
}

@article{nguyen2024_evo,
  title   = {Sequence modeling and design from molecular to genome scale with {Evo}},
  author  = {Nguyen, Eric and Poli, Michael and Durrant, Matthew G. and Kang, Brian and Katrekar, Dhruva and Li, David B. and Bartie, Liam J. and Thomas, Armin W. and King, Samuel H. and Brixi, Garyk and Sullivan, Jeremy and Ng, Madelena Y. and Lewis, Ashley and Lou, Aaron and Ermon, Stefano and Baccus, Stephen A. and Hernandez-Boussard, Tina and {R{\'e}}, Christopher and Hsu, Patrick D. and Hie, Brian L.},
  journal = {Science},
  volume  = {386},
  number  = {6723},
  pages   = {eado9336},
  year    = {2024},
  doi     = {10.1126/science.ado9336}
}

Credits

Original Evo / Evo 1.5 model and code by Nguyen et al. and Merchant et al. Source repo: evo-design/evo. Source checkpoint: evo-design/evo-1.5-8k-base.

The HuggingFace conversion code in this repo was authored primarily by Claude and reviewed manually by Taykhoom Dalal.

License

Apache 2.0, following the original Evo1 release.