FishCaduceus-28L-1024 is a fish-specific DNA language model in the FishCaduceus family. It was pretrained on fish genomic sequences using a masked language modeling objective and is intended for nucleotide-level representation learning, transfer learning, and sequence-based analyses in fish genomics.

This repository contains the pretrained FishCaduceus-28L-1024 checkpoint.

Model description

FishCaduceus is a family of Caduceus-based DNA language models developed for fish genomes. The models operate at single-nucleotide resolution and use bidirectional state-space sequence modeling together with reverse-complement-aware components.

The model learns to recover masked nucleotides from their surrounding genomic context. Its hidden representations can also be used as input features for downstream fish genomics tasks.

Model family

Model	Number of layers	Hidden dimension	Pretraining context length
FishCaduceus-20L-512	20	384	512 nt
FishCaduceus-28L-512	28	768	512 nt
FishCaduceus-28L-1024	28	768	1024 nt

Architecture

Backbone: Caduceus-based bidirectional state-space DNA language model
Training objective: masked language modeling
Tokenization: single-nucleotide character-level tokenization
Vocabulary size: 8
Number of layers: 28
Hidden dimension (d_model): 768
Pretraining context length: 1024 nucleotides
Selected training checkpoint: step 45,000
Framework: PyTorch and Hugging Face Transformers
Model type: custom Transformers model (trust_remote_code=True)

The context length above describes the sequence length used during pretraining. Users should apply truncation or windowing appropriate for the selected model variant.

Training data

The model was pretrained on curated fish genomic sequences obtained from publicly available genome resources. The genomes were processed through a unified quality-control and sequence-preparation workflow before construction of the masked-language-modeling corpus.

Detailed information about the included fish species, genome assemblies, preprocessing criteria, and dataset splits will be provided in the accompanying FishCaduceus dataset card and manuscript.

Training objective

FishCaduceus was pretrained using masked language modeling. Selected nucleotide positions were masked, and the model was optimized to predict the original nucleotide from the surrounding sequence context.

Because the tokenizer operates at single-nucleotide resolution, model outputs can be used for nucleotide-level probability estimation and sequence representation learning.

Intended uses

The model is intended for research applications including:

nucleotide-level genomic sequence representation learning
masked nucleotide prediction
extraction of frozen sequence embeddings
transfer learning and downstream fine-tuning
fish genomic annotation tasks
comparative and evolutionary genomics
exploratory variant-effect and regulatory-sequence analyses

Out-of-scope uses

The model is not intended to:

provide clinical or diagnostic decisions
replace experimental validation
establish biological causality from sequence scores alone
be used as the sole basis for decisions affecting human or animal health

Limitations

The model was developed primarily from fish genomic sequences, so performance may be lower for distant taxonomic groups.
Predictions reflect patterns learned from the pretraining corpus and do not by themselves demonstrate biological function or causality.
Performance may vary across species, genomic regions, sequence lengths, and downstream task definitions.
Long genomic sequences should be divided into windows compatible with the selected model's pretraining context length.
This repository contains custom modeling code. Review the repository code before loading it with trust_remote_code=True.

How to use

Install the dependencies listed in requirements.txt before loading the model.

Load the tokenizer and model

python

from transformers import AutoModelForMaskedLM, AutoTokenizer

model_id = "FishCaduceus/FishCaduceus-28L-1024"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model = AutoModelForMaskedLM.from_pretrained(
    model_id,
    trust_remote_code=True,
)

model.eval()

Masked nucleotide prediction

python

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

model_id = "FishCaduceus/FishCaduceus-28L-1024"
sequence = "ACGTACGT[MASK]ACGTACGT"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)
model = AutoModelForMaskedLM.from_pretrained(
    model_id,
    trust_remote_code=True,
)
model.eval()

inputs = tokenizer(sequence, return_tensors="pt")

with torch.inference_mode():
    outputs = model(**inputs)

mask_rows, mask_cols = (
    inputs["input_ids"] == tokenizer.mask_token_id
).nonzero(as_tuple=True)

base_tokens = ["A", "C", "G", "T"]
base_ids = tokenizer.convert_tokens_to_ids(base_tokens)

mask_logits = outputs.logits[mask_rows[0], mask_cols[0], base_ids]
base_probabilities = torch.softmax(mask_logits, dim=-1)

print(dict(zip(base_tokens, base_probabilities.tolist())))

Extract sequence representations

python

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

model_id = "FishCaduceus/FishCaduceus-28L-1024"
sequence = "ACGTACGTACGTACGT"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)
model = AutoModelForMaskedLM.from_pretrained(
    model_id,
    trust_remote_code=True,
)
model.eval()

inputs = tokenizer(
    sequence,
    return_tensors="pt",
    truncation=True,
    max_length=1024,
)

with torch.inference_mode():
    outputs = model(
        **inputs,
        output_hidden_states=True,
    )

token_embeddings = outputs.hidden_states[-1]
sequence_embedding = token_embeddings.mean(dim=1)

print("Token embeddings:", token_embeddings.shape)
print("Sequence embedding:", sequence_embedding.shape)

Repository files

The repository includes the files required to load the custom FishCaduceus model and tokenizer:

config.json
model weights (pytorch_model.bin or model.safetensors)
configuration_caduceus.py
modeling_caduceus.py
modeling_rcps.py
tokenization_caduceus.py
tokenizer.json
tokenizer_config.json
special_tokens_map.json
requirements.txt
README.md

Citation

The FishCaduceus manuscript is in preparation. Citation information will be added after publication.

Users of the model should also acknowledge the original Caduceus architecture and other upstream software components where appropriate.

Acknowledgements

FishCaduceus builds on the Caduceus architecture and related open-source sequence-modeling software. The model was developed for research on fish genomes at the Institute of Hydrobiology, Chinese Academy of Sciences.

License

This repository is released under the Apache License 2.0. Users are responsible for complying with the licenses of all upstream software dependencies.

Contact

For questions, please contact: xqxia@ihb.ac.cn