A LoRA fine-tune of Qwen/Qwen3.8-27B distilled on reasoning traces from Claude Opus 4.6/4.7, targeting the deep, structured chain-of-thought style Opus is known for — mapped onto an open-weight dense 27B you can run and fine-tune yourself.
Trained text-only (vision tower untouched, not targeted by LoRA) with Unsloth on a single H100.
Qwen3.8-27B is natively vision-language with a hybrid attention stack (16/64 full-attention layers, 48/64 Gated DeltaNet/linear-attention layers). This fine-tune targets only the text reasoning path — both attention types via LoRA, MLP layers, vision tower left frozen and unused.
Base model:Qwen/Qwen3.8-27B (dense, 27B, hybrid linear+full attention, MTP head)
Method: LoRA (r=32, alpha=32) via Unsloth, 4-bit QLoRA base
Response-only masking: loss computed only on assistant turns (train_on_responses_only)
Hardware: 1× NVIDIA H100 NVL (95GB), Vast.ai
Training data: ~21.5K examples combining real Opus extended-thinking traces and reconstructed-reasoning traces over genuine Opus outputs (see Dataset Composition below)
📊 Training Details
Steps trained
150 (partial run — pipeline validation, not full convergence)
Epochs
0.126 (~12.6% of one epoch, ~2,700 of 21,490 examples)
Effective batch size
18 (per-device 6 × grad accum 3)
Final train loss (avg)
0.728
Final eval loss
not computed this run (eval_strategy="no" to save time)
Wall-clock time
53 min on 1× H100 NVL
Sequence length
8192 (99.7% of dataset uncut; 0.3% longest examples excluded rather than truncated)
Trainable params
233,455,616 (0.85% of 27.6B)
Adapter size on disk
954 MB
Status: pipeline-validation run, not a fully converged model. 150 steps at batch 18 covers ~12.6% of one epoch — enough to confirm the training pipeline (LoRA on hybrid attention, masking, chat template, checkpointing) works end-to-end, not enough for the model to have generalized across the full dataset. A full run (1+ epoch, ~1,194+ steps) is needed before this should be treated as a finished distillation. Loss trended downward over the run (high ~0.95-1.1 early, several sub-0.6 readings by step 70-90) but with a cosine schedule calibrated for exactly 150 steps — this checkpoint cannot simply be resumed for a longer run; a full run needs a fresh schedule sized to the full dataset.
Real Opus 4.7 outputs (teacher_model field), reasoning trace reconstructed post-hoc by a smaller "trace inversion" model — plausible CoT matching a genuine answer, not captured original thinking
Transparency note: ~62% of the dataset (the two TraceInversion sets) has a genuine final answer from Opus but a reconstructed reasoning trace, not Opus's actual internal thinking. This is disclosed so downstream users know the reasoning style is a plausible approximation for roughly two-thirds of the data, not a verbatim mirror throughout. The lordx64 subset (~38%) is the closest to authentic Opus thinking style.
We deliberately excluded several other "Claude-distilled" datasets found on Hugging Face after inspection — some contained self-referential synthetic text describing itself as "exemplifying" a model's style rather than real model output, and others were low-effort duplicate repos farming downloads. Always verify teacher_model/usage provenance fields before trusting a distillation dataset.
📈 Base Model Benchmarks (Qwen/Qwen3.8-27B, not this fine-tune)
These are official Qwen3.8-27B numbers for the base model, included for reference — this fine-tune has not been independently re-benchmarked yet.
Text Performance
Qwen3.8-27B
Qwen3.6-27B
Qwen3.7-Plus
Terminal-Bench 2.1 (agentic coding)
73.0
63.4
64.0
SWE-bench Pro
61.7
53.5
57.6
QwenSWEBench
79.0
49.3
59.2
CoWorkBench (long-horizon office work)
70.7
61.0
65.1
IFBench (instruction following)
79.5
69.1
79.1
GPQA Diamond (scientific reasoning)
89.2
87.8
90.3
HLE (multidisciplinary reasoning)
30.8
24.0
34.7
LiveCodeBench v6
90.3
83.9
89.6
VL Performance (base model only — this fine-tune does not target vision)
Thinking is on by default (<think>...</think> block before the final answer), matching the base Qwen3.8 chat template. Use reasoning_content as a separate message field if constructing multi-turn history manually — do not pack the thinking block into content, the chat template expects it separate.
⚠️ Known Limitations
This checkpoint is a pipeline-validation run (150 steps, ~12.6% of one epoch), not a fully converged fine-tune. Expect it to show the target <think> format and some stylistic shift, but not robust generalization across task domains. A full training run is needed for production use — see status note above.
Dataset partially composed of reconstructed (not captured) reasoning traces — see Dataset Composition above
Text-only fine-tune; vision tower is unmodified base weights, not evaluated
🙏 Acknowledgments
Training methodology based on the Jackrong fine-tuning guide. Thanks to lordx64 and Jackrong for the source reasoning datasets.
