Views
No views yet
Qwen/Qwen3-4B-Instruct-2507 on stat.ML arXiv text
with offline-pregenerated thoughts interleaved at chunk boundaries. This is the
r_train = 0.125 arm of a thought-density sweep; the companion model
JackHsieh/prestar-4B-reason-only-k8-lr1e-05-replay0.75-epochs2-bs32-step2432
is identical except that it trains at r_train = 0.03125 (4× lower thought density).Qwen/Qwen3-4B-Instruct-2507 tokenizer is bundled
unmodified.| Base model | Qwen/Qwen3-4B-Instruct-2507 (trained from the released weights, not from an intermediate) |
| Checkpoint | latest / step-002432 — final step of the run (2 epochs) |
| Run name | 4B-reason-only-k=8-r_train=0.125/lr=1e-05-replay=0.75-epochs=2-bs=32 |
| Saved | 2026-07-16 (UTC) |
| Export | DCP (8-way FSDP-sharded, fp32 master weights) → consolidated → cast to bf16 safetensors |
| Codebase | prestar @ d053989, prestar.tools.checkpoint_hub to-hf |
| W&B | project prestar, entity latent-thoughts, group 4B-reason-only-k=8-r_train=0.125 |
JackHsieh/statML-arxiv-40M-20M, train split: 9,728 documents,
each exactly 4,096 tokens.JackHsieh/32B-reason-only.rule-r-1.0-k-8.L-512.statml-arxiv, train
split: 9.94M thoughts pregenerated offline by Qwen3-32B under a reason-only prompt
template, capped at 512 tokens each. One thought is sampled per thoughtful chunk per
step.JackHsieh/dclm-replay.seq-4096.tokens-32B, first 262,144 sequences of
4,096 tokens. Replay is 75% of served documents, mixed in to limit drift from the
base model's general distribution.r=0.125, k=8)k = 8-token chunks. The first chunk of every document is
always thoughtless; across the dataset exactly round(r × N_nonfirst) of the remaining
chunks are marked thoughtful (a thought is prepended before that chunk is predicted),
sampled uniformly with seed 0. Here r_train = 0.125 (1/8) of non-first chunks carry a
thought. Evaluation used r = 0.03125 for both arms of the sweep, so eval numbers
are comparable across thought densities — only the training density differs.| Optimizer | AdamW, lr 1e-5, wd 0.01, β = (0.9, 0.95), ε 1e-8 |
| Schedule | cosine, 5% warmup ratio, decay to 0 |
| Grad clip | 1.0 (global norm) |
| Batch | 32 documents/step (global), 2 epochs → 2,432 steps |
| Precision | FSDP shard_grad_op; fp32 master weights, bf16 params, fp32 gradient reduce |
| Attention | FlashAttention-2, torch.compile enabled |
| Hardware | 8 GPUs, single node |
| Seed | 0 |
checkpoint_meta.json in this repo carries the original run identity (step,
ordinal, run name, world size, save timestamp).