The model emits reasoning inside <think>...</think> blocks, as the base model and the SFT data both do. It was trained as the starting checkpoint for research on compressing chain-of-thought traces into short sequences of abstract-token embeddings, but nothing about the checkpoint is specific to that use — it is an ordinary reasoning SFT of Qwen3-4B.

Usage

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "sagisch/Qwen3-4B-Dolci-Think-SFT-7B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto", device_map="auto")

messages = [{"role": "user", "content": "How many r's are in strawberry?"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

out = model.generate(inputs, max_new_tokens=2048, temperature=0.6, top_p=0.95, top_k=20)
print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

Use the Qwen3 thinking-mode sampling settings — temperature 0.6, top-p 0.95, top-k 20. Greedy decoding is explicitly discouraged for Qwen3 and tends to produce repetition collapse. Context length is 40,960 tokens (max_position_embeddings), inherited from the base model.

Training


Base model	Qwen/Qwen3-4B (4B params, 36 layers)
Data	`allenai/Dolci-Think-SFT-7B`, deterministic 600k-example train subset (20k held out for eval)
Objective	Cross-entropy on assistant tokens only (prompt and template tokens masked)
Steps	4,518 (1 epoch), ~6.0B tokens seen
Global batch	128 sequences, token-budget batching at 32,768 tokens/micro-batch
Max sequence length	32,768 (2,716 of 600k rows dropped as longer)
Optimizer	AdamW, lr 1e-5, cosine schedule, 3% warmup, β=(0.9, 0.95), wd 0.1, grad clip 1.0
Precision	fp32 master weights, bf16 autocast, SDPA attention, gradient checkpointing
Hardware	4× H100, DDP
Seed	1337

Held-out eval loss over training (20k-example split, 512 examples per eval):

Step	10	3200	3600	4000	4400	4518
eval loss	—	0.8462	0.8450	0.8444	0.8442	0.8442

Train loss started at 1.2502 (step 10) and ended near 0.77–0.85. The curve is flat over the last ~1,000 steps, so this is a converged single epoch rather than an early stop.

These weights are the run's best checkpoint by held-out loss, cast to bf16. Optimizer state is not included — this checkpoint is for inference and further fine-tuning from scratch-initialized optimizer state, not for resuming the original run.

Evaluation

No benchmark numbers are reported here yet. Loss above is held-out cross-entropy on the Dolci eval split, which is a language-modeling metric, not a task-accuracy metric — do not read it as a capability comparison against the base model.

Limitations

Single epoch on one SFT dataset, with no preference tuning or RL on top. It inherits the base model's limitations and adds Dolci-Think's distributional biases.
No safety tuning was performed beyond whatever the base model and dataset carry.
Not evaluated on standard reasoning benchmarks, so any claim of improvement over Qwen/Qwen3-4B is currently unsupported.

License

Apache-2.0, inherited from Qwen/Qwen3-4B. Dolci-Think-SFT-7B carries its own terms; see the dataset card.