Topology-Aware Knowledge Distillation from Qwen3-30B-A3B → 1.7B
Convergent Intelligence LLC: Research Division
What This Is
TopologicalQwen is a 1.7B parameter model distilled from Qwen3-30B-A3B using Topological Knowledge Distillation (TKD) — a methodology that treats the teacher's output distribution over a concatenated token stream as a bounded variation (BV) function and decomposes knowledge transfer into three channels via the Mesh Fundamental Identity:
Smooth distillation (AC component) — Standard KL divergence over regions where the teacher's distribution varies continuously. This is what every other KD method does and stops at.
Jump corrections (D^j f) — Explicit correction terms at conceptual boundaries where the teacher's distribution exhibits discontinuities. These are the points where topic, register, or reasoning mode shifts — standard KD smears across them, losing structural information.
Drift corrections (D^c f) — The Cantor/singular-continuous component capturing gradual distributional drift that neither the smooth nor jump terms account for. This is the residual structure that emerges in generation quality.
Standard knowledge distillation only handles term (1). TKD captures all three.
DualMind format — each training sample is restructured into <explore> (derivation), <examine> (verification/self-critique), and <response> (clean answer) blocks. The model learns a cognitive loop: generate reasoning, then critique it, then synthesize.
TKD Pipeline (4 phases)
Phase 1 — Teacher logit caching: Single forward pass through the 30B teacher with top-64 logit compression to disk. One pass, no repeated teacher inference.
Phase 2 — DISC topology pass: Vectorized discrepancy operator maps the knowledge manifold. Jump detection at 3σ threshold with 1.25× amplification. Gap energy density computed over 64-token windows.
Phase 3 — Topology-guided adaptive windowing: 512-token windows cut at low-discrepancy positions (overlap 32–128) rather than fixed stride. The topology tells you where to cut without losing information across boundaries.
Phase 4 — Curriculum-ordered continuous KD: 4-phase curriculum (easiest 30% first). Proof-weighted loss: 2.25× → 1.1× decaying weights on reasoning tokens. KD alpha ramps from 0 → 0.45 (starting at 15% of training, reaching target at 45%). KL divergence at T=2.0. Effective batch size 32 (2 × 16 grad accumulation). Cosine LR: 5e-6 → 5e-7.
The model responds in DualMind format: <explore> → <examine> → <response>.
python
1from transformers import AutoModelForCausalLM, AutoTokenizer
23model = AutoModelForCausalLM.from_pretrained(4"reaperdoesntknow/TopologicalQwen",5 torch_dtype="auto",6 device_map="auto"7)8tokenizer = AutoTokenizer.from_pretrained("reaperdoesntknow/TopologicalQwen")910# Prompt with DualMind format — start the explore block11prompt =(12"##USER:\n"13"Prove that every convergent sequence is a Cauchy sequence.\n\n"14"<explore>\n"15)1617inputs = tokenizer(prompt, return_tensors="pt").to(model.device)18output = model.generate(19**inputs, max_new_tokens=2048, do_sample=True,20 top_p=0.9, temperature=0.6, repetition_penalty=1.1521)22result = tokenizer.decode(output[0], skip_special_tokens=True)23print(result)2425# Verify mode transitions26assert"<explore>"in result and"</explore>"in result # derivation27assert"<examine>"in result and"</examine>"in result # self-critique28assert"<response>"in result and"</response>"in result # clean answer
What the Output Looks Like
<explore>
[Unconstrained derivation — the model works through the proof freely]
</explore>
<examine>
[Adversarial self-response — the model critiques its own derivation]
</examine>
<response>
[Clean final answer synthesized from the internal dialogue]
</response>
This is the multi-model collision array collapsed into a single architecture. The dialectical structure that produces novel insights from architectural diversity is recreated through role-conditioned generation on shared weights.
Distillation Chain
Qwen3-1.7B (base)
→ DiStil-Qwen3-1.7B-uncensored (uncensored SFT)
→ Disctil-Qwen3-1.7B (DISC refinement)
→ TopologicalQwen (TKD from 30B-Thinking teacher + DualMind format) ← you are here
What Makes This Different
The broader Convergent Intelligence portfolio (open-weight portfolio) was trained on CPU at FP32 for a total compute cost of $24. That proves the methodology — structure beats scale.
This model is the exception. TopologicalQwen was trained on Colab H100 at BF16 precision with a 30B-parameter teacher. Same TKD methodology, premium compute. This is the DistilQwen collection's answer to "what happens when you give this pipeline real hardware?"
The result: a 1.7B model that exhibits dual-mental-modality reasoning (explore → examine → respond) with structural quality that standard distillation at any precision doesn't produce. The methodology is the constant. The hardware is the variable. Both produce results that shouldn't exist at this parameter count.
Every knowledge distillation method in the literature treats the teacher's output as a smooth function and minimizes KL divergence globally. This works for the easy parts — regions where the teacher's distribution varies slowly. But language has structure: topic shifts, reasoning mode transitions, register changes. At these boundaries, the teacher's distribution jumps. Standard KD averages across these jumps, teaching the student a blurred version of the teacher's actual knowledge.
TKD uses the DISC (Discrepancy Calculus) framework to detect these structural features before training, then allocates capacity and loss weight accordingly. The result is a student that preserves the teacher's structural understanding, not just its surface statistics.
The empirical evidence: this model at 1.7B consistently produces responses with structural reasoning quality that standard distillation at the same parameter count does not achieve.
TKD is grounded in Discrepancy Calculus — a measure-theoretic framework that treats singularities as primary structure rather than pathology. The full theory is developed in "On the Formal Analysis of Discrepancy Calculus" (CIx, 2026; Convergent Intelligence LLC: Research Division).
The Core Operator. The discrepancy operator quantifies local mismatch between integration and differentiation:
This is the theoretical backbone of TKD. Standard knowledge distillation captures only the first term. TKD preserves all three.
TKD Application. The teacher's output distribution $p_T(x)$ over a concatenated token stream is treated as a BV function. The DISC topology pass computes:
Discrepancy energy $E_{\text{disc}}[p_T] = \frac{1}{2}\int w(x)(Dp_T(x))^2 dx$ — identifies regions of high structural information density
Gap energy density over 64-token windows — measures Cantor-type drift invisible to both smooth and jump analysis
Windows are cut at low-discrepancy positions rather than fixed stride. Loss weight is amplified at jump positions (1.25×). The topology tells you where the knowledge has architecture.
Why This Matters (Meta-Discrepancy Theorem). Theorem 11.15 of the DISC monograph proves: when the gap measure $\mu_{\text{gap}} > 0$ and discrepancy energy $E_{\text{disc}} > 0$, the classical FTC/MVT/chain-rule package is impossible on positive measure. Standard KD — which implicitly assumes smooth teacher distributions — provably cannot capture the structural information that TKD preserves. This is not a heuristic argument. It is a mathematical impossibility result.
DistilQwen Collection — Our only BF16 series. Proof-weighted distillation from Qwen3-30B-A3B → 1.7B and 0.6B on H100. Three teacher variants (Instruct, Thinking, Coder), nine models. The rest of the portfolio proves structure beats scale on CPU. This collection shows what happens when you give the methodology real hardware.
DualMind Collection — Dual cognition architecture. Single model, two internal voices, three cognitive phases. Five models including Dualmind-Qwen-1.7B-Thinking (Opus 4.6 reasoning variant).
