| Component | Configuration |
|---|---|
| Parameters | 599.9M |
| Layers | 18 |
| Hidden Size | 1280 |
| Attention Heads | 16 |
| Head Dimension | 80 |
| Intermediate Size | 5120 (SwiGLU) |
| Vocabulary | 50,000 (SentencePiece) |
| Context Length | 2048 |
| Positional Encoding | RoPE (θ=10000) |
| Normalization | RMSNorm (pre-norm) |
| Metric | Value |
|---|---|
| Total Tokens | 39.32B |
| Training Steps | 300,000 |
| Batch Size | 256 (global) |
| Learning Rate | 3e-4 → 3e-5 (cosine decay) |
| Hardware | TPU v5litepod-32 |
| Framework | JAX + Flax |
| Precision | bfloat16 |
| Final Loss | 2.33 |
| Final Perplexity | 10.3 |
| Source | Proportion | Tokens |
|---|---|---|
| Wikipedia EN | ~25% | ~10B |
| Wikipedia FR | ~10% | ~4B |
| OSCAR (EN/FR) | ~40% | ~16B |
| The Stack (Code) | ~15% | ~6B |
| Gutenberg Books | ~10% | ~4B |
| Benchmark | Score |
|---|---|
| HellaSwag | 53.5% |
| PIQA | 66.8% |
| LAMBADA | 37.3% |
| Model | Params | Tokens | HellaSwag | PIQA | Year |
|---|---|---|---|---|---|
| SmolLM2-135M | 135M | 2T | 43.3% | 67.4% | 2025 |
| Pythia-410M | 410M | 300B | 40.9% | 66.8% | 2023 |
| SmolLM2-360M | 360M | 4T | 54.5% | 71.7% | 2025 |
| Qwen2.5-0.5B | 490M | ~18T | 52.1% | 69.9% | 2024 |
| Julian 600M | 600M | 39B | 53.5% | 66.8% | 2025 |
| Qwen3-0.6B-Base | 600M | 36T | 41.1% | 70.0% | 2025 |
| Pythia-1B | 1B | 300B | 49.7% | 70.7% | 2023 |
| SmolLM2-1.7B | 1.7B | 11T | 68.7% | 76.9% | 2025 |
💡 Key insight: With only 39B tokens (ratio 1:65), Julian 600M surpasses Qwen3-0.6B-Base (41.1% HellaSwag, trained on 36T tokens, same 600M params) and matches SmolLM2-360M (54.5%, trained on 4T tokens). Julian achieves this with 100-900x less training data, highlighting exceptional data efficiency.
1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3model = AutoModelForCausalLM.from_pretrained("JulianKrgd/julian-600m-40b")
4tokenizer = AutoTokenizer.from_pretrained("JulianKrgd/julian-600m-40b")
5
6prompt = "La France est"
7inputs = tokenizer(prompt, return_tensors="pt")
8outputs = model.generate(**inputs, max_new_tokens=100)
9print(tokenizer.decode(outputs[0], skip_special_tokens=True))1import jax
2from julian_model import JulianLM, JULIAN_600M
3import orbax.checkpoint as ocp
4
5# Load model
6model = JulianLM(JULIAN_600M)
7checkpointer = ocp.PyTreeCheckpointer()
8params = checkpointer.restore("path/to/checkpoint")
9
10# Generate
11# See src/inference/generate.py for full example| Step | Tokens | Loss | Perplexity |
|---|---|---|---|
| 0 | 0 | 10.5 | 36,316 |
| 50K | 6.5B | 3.20 | 24.5 |
| 100K | 13.1B | 2.70 | 14.9 |
| 150K | 19.6B | 2.50 | 12.2 |
| 200K | 26.2B | 2.40 | 11.0 |
| 250K | 32.8B | 2.35 | 10.5 |
| 300K | 39.3B | 2.33 | 10.3 |
| Metric | Value |
|---|---|
| TPU Type | v5litepod-32 |
| TPU Hours | ~120h |
| Total FLOPS | ~2.4e19 |
| Throughput | ~1.1M tok/s |
| Training Time | 5 days |
1# Hyperparameters
2learning_rate: 3e-4 → 3e-5 (cosine decay)
3warmup_steps: 3000
4batch_size: 256 (global)
5sequence_length: 2048
6weight_decay: 0.1
7gradient_clipping: 1.0
8precision: bfloat16
9
10# Optimizer
11optimizer: AdamW
12beta1: 0.9
13beta2: 0.95
14epsilon: 1e-8| Model | Type | Training | HellaSwag | PIQA | LAMBADA | Status |
|---|---|---|---|---|---|---|
| julian-600m-10b | Base | 10B tokens | 45.8% | 67.6% | 35.0% | ✅ Released |
| julian-600m-40b | Base | 39B tokens | 53.5% | 66.8% | 37.3% | ✅ Current |
| julian-600m-10b-instruct-v0.1 | SFT | 10B + 185K ex | 42.7% | 66.2% | 34.6% | ✅ Released |
| julian-600m-40b-sft-2.5M | SFT | 39B + 2.5M ex | ~ | ~ | ~ | 🔄 Coming |
1@misc{julian2025,
2 author = {Julian Kerignard},
3 title = {Julian: A 600M Parameter Language Model Trained on 40B Tokens},
4 year = {2025},
5 publisher = {Hugging Face},
6 url = {https://huggingface.co/JulianKrgd/julian-600m-40b}
7}