You can deploy this model and start earning money today!

OpenRaiser/
fwe_deltanet_1b_lion_lr3e_3_b1_0_9_b2_0_99

Name: fwe_deltanet_1b_lion_lr3e_3_b1_0_9_b2_0_99
Availability: InStock

Views

No views yet

fwe_deltanet_1b_lion_lr3e_3_b1_0_9_b2_0_99_20260520_105611

This is a 1B DeltaNet language model trained on FineWeb-Edu using the Lion optimizer, as part of the OpenTome optimizer benchmark.

Model Details

Field	Value
Architecture	DeltaNet
Model Type	`delta_net`
Scale	1B
Hidden Size	2048
Layers	24
Attention Heads	16
Vocab Size	32000
Max Sequence Length	2048
Training Dataset	FineWeb-Edu
Optimizer	Lion
Hyperparameters	lr=3e-3, β₁=0.9, β₂=0.99
Timestamp	20260520_105611

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "OpenRaiser/fwe_deltanet_1b_lion_lr3e_3_b1_0_9_b2_0_99_20260520_105611",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    "OpenRaiser/fwe_deltanet_1b_lion_lr3e_3_b1_0_9_b2_0_99_20260520_105611",
    trust_remote_code=True,
)

Training

See train.sh for the full training command. Training was conducted on H200 GPUs using the FLAME framework.

Citation

If you use this model, please cite the OpenTome benchmark paper.