CENO is derived from NVIDIA's Nemotron-H (Apache-2.0). The custom Transformers remote code in this repository (configuration_ceno.py, modeling_ceno.py) is a rename of the upstream Nemotron-H implementation.

This repository includes custom Transformers remote code for CENOForCausalLM and CENOCharLevelTokenizer. Load with trust_remote_code=True.

Files

model.safetensors: model weights
config.json: model config with auto_map
generation_config.json: generation config
configuration_ceno.py, modeling_ceno.py: custom model code
ceno_tokenizer.py, tokenizer_config.json, special_tokens_map.json, vocab.json: tokenizer files
training_metrics.json: finetuning metrics

Loading

python

from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "CladeTeam/CENO-rice-cds"
model = AutoModelForCausalLM.from_pretrained(repo_id, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)

The model code depends on PyTorch and the Mamba/Triton stack used by Nemotron-H. The bundled config sets use_mamba_kernels=false, using the pure-PyTorch Mamba fallback so no mamba-ssm/causal-conv1d install is required.

Training

Finetuned for 1 epoch on rice CDS with learning_rate=5e-5, effective_batch_size=64, bf16, max_length=8192.

Training Metrics

json

{
  "species": "rice",
  "train_loss": 10.05208391170438,
  "eval_loss": 1.21553373336792,
  "learning_rate": 5e-05,
  "epochs": 1,
  "epoch_losses": [
    {
      "epoch": 0.9987473903966597,
      "eval_loss": 1.21553373336792
    },
    {
      "epoch": 0.9987473903966597,
      "eval_loss": 1.21553373336792
    }
  ],
  "n_gpu": 8,
  "effective_batch_size": 64
}

Intended Use

These models are released to reproduce HTT/polyQ sequence scoring experiments. The average log-likelihood scores reflect sequence-model likelihood, not biological fitness or pathogenicity.

License

This model and its bundled code are released under the Apache License 2.0, inheriting the license of the upstream Nemotron-H model code (Copyright 2024 AI21 Labs Ltd. and the HuggingFace Inc. team; Copyright (c) 2025 NVIDIA CORPORATION). Modifications for CENO by CladeTeam.