Aris-375M-v0.1
Aris-375M-v0.1 is a 375 million parameter decoder-only language model trained from scratch using the Aris 96K tokenizer.
This release serves as the initial public research checkpoint of the Aris model family.
Model Details
- Parameters: ~375M
- Architecture: Llama-style causal decoder
- Tokenizer: joseph-ai/Aris-Tokenizer-96K
- Context length: 4096
- Training: packed-token pretraining
- Training tokens: approximately 5B
Evaluation
The model was evaluated on FineWeb-Edu validation data and standard zero-shot language model benchmarks.
Zero-shot lm-eval results:
| Task | Metric | Score |
|---|
| ARC-Challenge | acc_norm | 0.2696 |
| ARC-Easy | acc | 0.5383 |
| HellaSwag | acc_norm | 0.3314 |
| WinoGrande | acc | 0.5067 |
Intended Use
This model is intended for research into small language model pretraining, data mixture design, and tokenizer design.
Limitations
This is a base model and has not been instruction tuned. It may produce incorrect, repetitive, biased, or unsafe text. It should not be used for production decision-making or user-facing assistant behavior without further safety and alignment training.
Training Procedure
The model was trained with packed sequences using multi-GPU bf16 training.
Key hyperparameters from the final training stage:
- learning_rate: 5e-05
- train_batch_size: 2
- eval_batch_size: 2
- num_devices: 2
- gradient_accumulation_steps: 16
- total_train_batch_size: 64
- optimizer: AdamW fused
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 100
Framework Versions
- Transformers 5.12.1
- PyTorch 2.4.1+cu124
- Datasets 5.0.0
- Tokenizers 0.22.2
