You can deploy this model and start earning money today!

YifeiZuo/
attention-0.6b-adamw-wsd-step4000

Name: attention-0.6b-adamw-wsd-step4000
Availability: InStock

pretraining-checkpoint

Qwen/Qwen3-0.6B

finetune

apache-2.0

Views

No views yet

Attention 0.6B AdamW-WSD - step 4000

Intermediate checkpoint at optimizer step 4000 (of 20000) of the 0.6B Qwen3-style softmax-attention baseline used in the Parallax mechanism analysis.

Architecture: Qwen3-0.6B (28 layers, d_model=1024, 16 Q-heads, 8 KV-heads, GQA, RoPE theta 1e6)
Optimizer: AdamW (lr=3e-4, weight decay=0.1, betas=(0.9, 0.95))
Scheduler: WSD (warmup=0, last 20% linearly decayed)
Tokens seen at this step: ~15720.0M sequences * 4096 tokens = ~62.9B tokens
Dataset: Ultra-FineWeb (English split)
Tokenizer: Qwen3-0.6B (vocab 151936)
Total steps in run: 20000 (~80B tokens)

The full training trajectory (every 500 steps) lives in the Attention 0.6B AdamW-WSD training trajectory collection.

Loading

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "YifeiZuo/attention-0.6b-adamw-wsd-step4000",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("YifeiZuo/attention-0.6b-adamw-wsd-step4000")