Views
No views yet
| Property | Value |
|---|---|
| Architecture | DaisyCore |
| Layers | 26 |
| Attention Heads | 14 |
| Model Dimension | 1,792 |
| Head Dimension | 128 |
| Sliding Window Size | 2,048 |
| Max Sequence Length | 131,072 |
| Vocabulary Size | 49,152 |
| Attention Implementation | standard |
| Value Embeddings | True |
| Tied Embeddings | False |
| Skip Mix Mode | linear |
| Tokenizer | jonathanmiddleton/daisy |
| Dtype | bfloat16 |
| Parameters (total) | 2,323,120,245 |
| Parameters (non-embedding) | 1,001,914,485 |
| Parameters (embedding) | 1,321,205,760 |
| Metric | Value |
|---|---|
| Checkpoint Step | 2,058 |
| Tokens Processed | 3.37B (3,371,827,200) |
| Target Tokens | 3.37B (3,371,827,200) |
| Progress | 100.0% |
| Best Validation Loss | 1.41598 |
| Evaluations Performed | 11 |
| Saved | 2026-03-02 23:10 UTC |
| Optimizer | Parameter Group | Learning Rate |
|---|---|---|
| AdamW | head_params | 0.003216 |
| AdamW | embed_params | 0.1865 |
| AdamW | scalar_params | 0.02099 |
| Muon | hidden_matrix_params | 0.025 |
| Parameter | Value |
|---|---|
| LR Scale | 1.0 |
| LR Schedule | n_phase_linear |
| LR Schedule — begin_after_fraction | 0.0 |
| LR Schedule — cooldown_fraction | 0.0 |
| LR Schedule — floor | 0.0 |
| LR Schedule — phases | [{'progress': 0.0, 'scale': 0.1}, {'progress': 0.3, 'scale': 0.1}, {'progress': 1.0, 'scale': 0.01}] |
| LR Schedule — warmup_fraction | 0.0 |
| Gradient Accumulation Steps | 4 |
| Muon Warmup Steps | 300 |
| Seed | 1337 |
| Type | Sequence Length | Path |
|---|---|---|
| fineweb-edu-dedup | 8,192 | data/fineweb-edu-dedup/fineweb-edu-dedup_jonathanmiddleton_daisy_train_*.bin[000160:000200] |
| daisy_assistant_v1 | 1,024 | data/daisy_assistant_v1 |
| daisypie_smart | 16,384 | data/daisypie_smart |
| Parameter | Value |
|---|---|
| window_size | 2048 |
| vocab_size | 49152 |
| eos_token_id | 49131 |
| num_layers | 26 |
| num_heads | 14 |
| model_dim | 1792 |
| head_dim | 128 |
| max_seq_len | 131072 |
| model_spec | daisy_milli |
| model_class | models.daisy.daisy_core.DaisyCore |
| target_tokens | 3371827200 |
| full_window_target_tokens | 3371827200 |
| torch_coordinate_descent_tuning | False |
| torch_inductor_config_max_autotune | False |
| overfit | False |
| full_windows | True |
| wandb_log | True |
| wandb_project | milli |
| wandb_run_name | milli_v18b.13.1 |
| wandb_group | sft |
| init_model | JonathanMiddleton/daisy-milli-18b.13 |
| use_value_embeddings | True |
| use_tied_embeddings | False |
| seed | 1337 |
| task_val_debug_log_samples | False |
| log_interval | 16384 |
| muon_warmup_steps | 300 |
| lr_scale | 1.0 |
| cooldown_fraction | 0.0 |
| lr_schedule | {"name": "n_phase_linear", "config": {"cooldown_fraction": 0.0, "phases": [{"progress": 0.0, "scale": 0.1}, {"progress": 0.3, "scale": 0.1}, {"progress": 1.0, "scale": 0.01}], "floor": 0.0, "warmup_fraction": 0.0, "begin_after_fraction": 0.0}} |
| grad_acc_steps | 4 |
| val_loss_every_tokens | 327680000 |
| checkpoint_warmup_tokens | 1000000000 |
| checkpoint_per_n_tokens | 0 |
| save_checkpoint | True |
| benchmarks_frequency | 1 |
| mmlu_cache_bin_path | data/mmlu_cache/mmlu_cache.bin |
| mmlu_cache_bin_rebuild | False |
| task_training | False |
| track_last_n_layers | 0 |