Titanius-1 is an experimental decoder-only base language model. This preview is the result of a default training run with no command-line parameter overrides, stopped at the nominal 7,000-step checkpoint.
Preview status: This is a research artifact, not an instruction-tuned or production-ready assistant. It may generate inaccurate, biased, unsafe, or otherwise undesirable text.
Training overview
Training loss, learning rate, and gradient norm over 7,000 steps
Setting
Value
Training steps
7,000
Estimated training tokens
458,752,000
Tokens per optimizer step
65,536
Context length
2,048 tokens
Device batch size
1 sequence
Gradient accumulation
32 steps
Optimizer
AdamW
Peak learning rate
4e-4
Final learning rate
4e-5
Warmup
2%
Schedule
Cosine decay
Gradient clipping
1.0
Final smoothed training loss
3.0718
Training dataset
karpathy/climbmix-400b-shuffle
The token estimate is 7,000 × 65,536. No held-out validation or standardized benchmark evaluation was recorded for this preview, so training loss should not be interpreted as a measure of downstream quality.
Model architecture
Component
Value
Architecture
Decoder-only causal Transformer
Layers
10
Hidden size
768
Query heads
12
Key/value heads
4
Head dimension
64
MLP width
3,072
Vocabulary size
32,768
Positional encoding
Rotary embeddings (RoPE)
Attention
Grouped-query causal attention
Activation
Squared ReLU
Context window
2,048 tokens
Output logit soft cap
15
The model uses RMS normalization in the forward path, QK normalization, untied input/output embeddings, per-layer residual scaling, an input-embedding residual, and a mid-layer “backout” residual.
Parameter count
The parameter count was recalculated directly from checkpoints/model_step_7000.pt:
Count
Parameters
Active parameters used by the forward pass
113,246,229
Stored checkpoint parameters
160,447,509
Instantiated but unused parameters
47,201,280
The repository’s “115m” label is a rounded name for the approximately 113.25M active-parameter model. The raw training checkpoint is larger because every attention module instantiates an additional MLP and LayerNorm that are not called by the model’s forward pass. Those tensors are nevertheless included in the PyTorch state dictionary. Consumers exporting or converting the model should account for this distinction.
Tokenizer
The model uses a custom byte-level BPE tokenizer with a vocabulary of 32,768 tokens. Numbers are pre-tokenized in groups of up to three digits. Its special tokens are:
Although conversation boundary tokens exist in the vocabulary, this checkpoint is a base pretrained model and has not been instruction-tuned.
Intended use
This checkpoint is intended for:
research and experimentation with small causal language models;
continued pretraining or supervised fine-tuning;
inspection of the custom architecture and training run;
non-critical text-generation experiments.
It is not intended for autonomous decisions, high-stakes applications, factual lookup, or deployment without application-specific evaluation and safeguards.
Limitations
This is an early preview trained for only 7,000 steps (about 459M tokens).
No benchmark, validation-set, safety, bias, or memorization evaluation is reported.
The model is not instruction-tuned and may not reliably follow prompts.
Outputs may be incorrect, incoherent, biased, offensive, or repetitive.
The training corpus may contain problematic, copyrighted, personal, or low-quality material inherited from its upstream sources.
Loading and inference require the repository’s custom PyTorch model code; this is not a drop-in standard Transformers architecture.
Training data
The run streams the training split of karpathy/climbmix-400b-shuffle. Users should review the upstream dataset card, composition, license information, and usage restrictions before using or redistributing this model.
Reproducibility notes
The default run configuration was read from the training code, while model dimensions and the final smoothed loss were verified against the saved 7,000-step checkpoint. The training plots above were generated from the accompanying CSV log. Hardware, exact compute dtype, random seed for model initialization, and independent evaluation results were not recorded in the checkpoint and are therefore not claimed here.
Citation
If you use this preview, cite the Hugging Face repository:
bibtex
1@misc{salem2026titanius1,
2 author = {Nadav Salem},
3 title = {Titanius-1-115m-base-fp16-preview},
4 year = {2026},
5 howpublished = {\url{https://huggingface.co/NadavSalem/Titanius-1-115m-base-fp16-preview}}
6}
