A 0.6B parameter model distilled from Qwen3-30B-A3B-Thinking on 6,122 STEM chain-of-thought samples. 50x parameter compression. The Thinking variant teacher produces richer extended reasoning traces than the Instruct variant, transferring deeper deliberation structure into the smallest possible student.
The result: a model under 500MB quantized that produces structured STEM derivations because a 30B thinking model showed it how to reason.
"Structure beats scale."
— Convergent Intelligence LLC: Research Division
What Makes This Different
Two key differences from standard small-model distillation:
1. Thinking teacher, not Instruct teacher. The Qwen3-30B-A3B-Thinking variant generates extended internal reasoning before committing to an answer. Its softmax distributions are higher-entropy — it considers more reasoning paths at each step. At distillation temperature T=2.0, this means the 0.6B student sees a much richer landscape of alternative derivation strategies than it would from an Instruct teacher. The student doesn't just learn the answer — it learns the deliberation.
2. Proof-weighted loss. Tokens inside the derivation region (Proof: to Final Answer:) receive 2.5x amplified loss, decaying to 1.5x over training. The model is penalized more for errors in reasoning steps than for errors in answer formatting. At 0.6B, every parameter has to count — proof weighting ensures they're allocated to reasoning capability, not boilerplate reproduction.
Solve the following problem carefully and show a rigorous derivation.
Problem:
{question}
Proof:
{CoT}
Final Answer:
{response}
Usage
python
1from transformers import AutoTokenizer, AutoModelForCausalLM
2import torch
34model_id ="reaperdoesntknow/Qwen3-0.6B-STEM-Proof-Distilled-Thinking"56tokenizer = AutoTokenizer.from_pretrained(model_id)7model = AutoModelForCausalLM.from_pretrained(8 model_id,9 torch_dtype=torch.bfloat16 if torch.cuda.is_available()else torch.float32,10 device_map="auto",11)1213prompt ="""Solve the following problem carefully and show a rigorous derivation.
1415Problem:
16Find the eigenvalues of the matrix [[3, 1], [0, 3]].
1718Proof:
19"""2021inputs = tokenizer(prompt, return_tensors="pt").to(model.device)22with torch.no_grad():23 outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)24print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Intended Uses
Good for: Lightweight STEM reasoning on edge/mobile devices, educational tutoring, proof drafting, component in multi-model pipelines where a small fast reasoner is needed, IoT and embedded inference.
Not for: Formal proof verification, safety-critical analysis, medical or legal advice, or tasks requiring long-context reasoning beyond 1024 tokens.
Limitations
0.6B is a hard capacity constraint. The model will struggle with multi-step proofs requiring more than ~8 reasoning steps, complex multi-variable problems, or domains underrepresented in training data (molecular biology, physiology). It will sometimes generate plausible but incorrect intermediate steps. Always verify.
This model is part of a distillation chain built on Discrepancy Calculus — a measure-theoretic framework where the teacher's output distribution is decomposed via the Mesh Fundamental Identity into smooth (AC), jump, and Cantor components. The discrepancy operator $Df(x) = \lim_{\varepsilon \downarrow 0} \frac{1}{\varepsilon} \int_x^{x+\varepsilon} \frac{|f(t) - f(x)|}{|t - x|} dt$ quantifies local structural mismatch that standard KL divergence averages away.
Full theory: "On the Formal Analysis of Discrepancy Calculus" (CIx, 2026; Convergent Intelligence LLC: Research Division). Full methodology: Structure Over Scale (DOI: 10.57967/hf/8165).
This model is part of a distillation chain built on Discrepancy Calculus — a measure-theoretic framework where the teacher's output distribution is decomposed via the Mesh Fundamental Identity into smooth (AC), jump, and Cantor components. The discrepancy operator $Df(x) = \lim_{\varepsilon \downarrow 0} \frac{1}{\varepsilon} \int_x^{x+\varepsilon} \frac{|f(t) - f(x)|}{|t - x|} dt$ quantifies local structural mismatch that standard KL divergence averages away.
Full theory: "On the Formal Analysis of Discrepancy Calculus" (CIx, 2026; Convergent Intelligence LLC: Research Division). Full methodology: Structure Over Scale (DOI: 10.57967/hf/8165).
The only BF16 collection in the portfolio. While the broader Convergent Intelligence catalog (43 models, 12,000+ downloads) was trained on CPU at FP32 for $24 total compute, the DistilQwen series was trained on H100 at BF16 with a 30B-parameter teacher. Same methodology, premium hardware. This is what happens when you give the pipeline real compute.
All models use proof-weighted knowledge distillation: 55% cross-entropy with decaying proof weights (2.5× → 1.5×), 45% KL divergence at T=2.0. The proof weight amplifies loss on reasoning-critical tokens, forcing the student to allocate capacity to structural understanding rather than surface-level pattern matching.
