Helios Nova 306M is a 306M-parameter, dense, decoder-only language model pre-trained from scratch on 50B tokens of FineWeb-Edu. It is the base model of the Helios Nova family; the architecture, tokenizer, pre-training, and evaluation were developed independently and end-to-end by a single author.
The model was built to study capability per unit of compute at small scale. At roughly 80× less pre-training data, it reaches 96% of SmolLM2-360M on commonsense reasoning (Winogrande + PIQA), measured on an identical evaluation harness. Pre-training cost under USD 190 of compute on a single GPU.
This is a base (pre-trained) model intended for text completion and as a starting point for fine-tuning. For instruction following and chat, use the instruction-tuned releases linked below.
Highlights
306M dense decoder, custom architecture and 16k tokenizer, trained from scratch.
Data-efficient: 96% of SmolLM2-360M commonsense reasoning at ~80× fewer pre-training tokens; ties it on Winogrande.
Low cost: 50B tokens on a single NVIDIA H100 in under 120 hours, for under USD 190.
Helios Nova 306M was pre-trained on 50B tokens of FineWeb-Edu on a single NVIDIA H100 in under 120 hours, for under USD 190. It uses a Warmup-Stable-Decay (WSD) learning-rate schedule with fused AdamW, bf16, and torch.compile. FineWeb-Edu (the educationally filtered subset of FineWeb) was chosen deliberately: the goal was to test whether architecture and a clean corpus could carry data efficiency at a fraction of the usual token budget. The validation loss decreases throughout the stable phase and drops sharply during the final decay.
All models below were re-run through one identical lm-evaluation-harness configuration (0-shot), so the comparison is internally consistent; these figures therefore differ slightly from each model's published numbers.
Capability versus pre-training token budget
Metric (0-shot)
Helios-306M (50B tok)
SmolLM2-360M (~4T)
Qwen2.5-0.5B (~18T)
Winogrande
57.2
57.9
56.3
PIQA
68.1
72.6
70.6
OpenBookQA
34.4
37.6
35.4
HellaSwag
44.7
52.5
49.5
ARC (avg)
42.8
53.4
45.5
MMLU
24.3
25.3
47.6
Commonsense reasoning (Winogrande + PIQA)
62.65
65.25
63.45
Helios reaches 96.0% of SmolLM2-360M on commonsense reasoning (Winogrande + PIQA) at roughly 80× less pre-training data, and ties it on Winogrande (99%). It trails on tasks bounded by data volume — broad factual recall (TriviaQA) and exam-style knowledge, where Qwen2.5-0.5B's much larger curated corpus is decisive. Helios Nova is data-efficient, not knowledge-rich.
Full benchmark sweep
Intended use and limitations
This is a base model: it performs next-token continuation and is intended for text completion and as a foundation for fine-tuning (instruction tuning, preference optimization, domain adaptation). It is not instruction-tuned and will not reliably follow prompts; for that, use the instruction-tuned releases below.
A 306M-parameter model trained on 50B tokens of educational text has limited world knowledge and performs near chance on broad factual recall (TriviaQA) and exam-style benchmarks (MMLU). Outputs may be inaccurate; verify before use. The model is English-only and has received no safety alignment.
GRPO-aligned instruction model (GGUF and safetensors)
Citation
bibtex
1@misc{espinosamena2026heliosnova,
2 title = {Helios Nova 306M: a data-efficient language model pre-trained from scratch on a single GPU},
3 author = {Espinosa Mena, Rafael},
4 year = {2026},
5 howpublished = {\url{https://huggingface.co/respinosamena/Helios-Nova-306M}}
6}
