Views
No views yet
magpie (90%) + delphi_warmup (10%) mixture, starting from the prepared 1e23 midtrained base.Qwen3ForCausalLM (dense), vocab 128256, 51 layers, hidden 5376,
42 attention / 42 KV heads, head_dim 128, max_position_embeddings 4096.rope_theta = 500000,
rope_scaling = {rope_type: llama3, factor: 8.0, low_freq_factor: 1.0, high_freq_factor: 4.0, original_max_position_embeddings: 8192} (mirrors the base; carried at both the top level and under
rope_parameters for transformers-4/5 compatibility).laion/delphi-1e23-p33m67-k0p076-prepared
(prepared from the midtrained laion/delphi-1e23-p33m67-k0p076-lr0.67-b6607e).magpie 90% / delphi_warmup 10%.<|start_header_id|>/<|end_header_id|>/<|eot_id|>) plus repurposed single-id think/tool tokens:| token | id |
|---|---|
| `< | start_think |
| `< | end_think |
| `< | tool_call |
| `< | tool_call_end |
| `< | tool_result |
| `< | tool_result_end |
<|im_start|>/<|im_end|> are absent from this tokenizer. The Delphi v0
Jinja template is not baked into tokenizer_config.json; apply it externally (e.g. pass it to the
serving engine). Reference implementation:
experiments/sft/delphi_chat_template.py
(the served template is that string with the Levanter {% generation %}/{% endgeneration %} markers
removed). When decoding, keep skip_special_tokens=False so the atomic 128002/128003 think delimiters
survive.