Qwopus3.6-27B Fusion — one 27B that reasons and codes, no skill lost
▶ PLAY ORBITAL — a game this model built, live in your browser ◀
Research preview. This model attempts to combine the reasoning capability of
Qwopus3.6-27B-v2 with the code-execution capability of Qwopus3.6-27B-Coder — in a single 27B
model, without significant loss to either skill. In practice it behaves like a production model
with both capabilities fused, but it has not yet been through a full rigorous evaluation. If you
find issues, please reach out on X — @kylehessling1.
Format: GGUF, Q4_K_M (~16.5 GB) — runs on a single 24–32 GB GPU via llama.cpp.
Modes: thinking (reasoning) on by default; strong agentic/coding behavior.
Vision: 🖼️ image input supported via the companion projector Qwopus3.6-27B-Fusion-mmproj-Q8_0.gguf (~0.6 GB) — pair it with any quant below (see Vision).
◆ Where it shines — the one-stop creative programming agent
Best used as a single, closed-loop agentic creative programmer. In a build → run → self-fix
loop, Fusion is at its strongest as one agent that plans, writes, and debugs on its own — the
reasoning⊕coding balance lets it close the loop by itself (see the ORBITAL / FLUX demos below,
where it root-caused and fixed its own bugs from Playwright error reports).
This is a different niche than its parents. The Coder model is excellent for coding under
active steering, or as a solid orchestration model driving other tools/agents. Fusion, by
contrast, is the ideal one-stop creative builder — you hand it a goal and it carries the whole
creative-programming loop end-to-end.
Two finetunes of the same base diverge into complementary specialists: one gets better at
multi-step reasoning, the other at writing code — but each pays for it. In particular the coding
finetune becomes unstable in thinking mode (it loops / degrades at low temperature). The goal was
a single model that keeps the reasoner's stability and the coder's skill.
My naive approaches, one of them eroneously proposed by Claude Fable, average the two, or run an equal-weight DARE-TIES merge from the common base. These alternatives underperformed and the geometry explains why.
▌ The lineage
Because the coder model descended from the v2 version, the coders additions can be evaluated against the v2 by analyzing the delta. Projecting the coding
delta onto the reasoning delta reveals:
proj(ΔCoder · ΔV2 / |ΔV2|²) ≈ 0.4 in early layers → ≈ 1.0 in late layers
That ramp to 1.0 means late layers of the coder already contain V2's full reasoning delta. And:
The coding delta is ~1.8× the magnitude of the reasoning delta.
In early layers the coding delta is anti-aligned with reasoning (cos < 0) — the coder
partially reverted V2's early-layer reasoning/control representations. That anti-alignment is
the measured source of the coder's thinking-mode instability.
An equal-weight merge (both task vectors added from the base) therefore commits two errors:
It double-counts V2's reasoning delta (the coder already contains it), and
It imports the full early-layer destabilizing coding delta at 100% strength.
▌ The fix
Because the coder sits "downstream" of V2, the correct anchor is V2 itself, not the shared base
— this avoids the double-count. Then, instead of a constant fraction of the coding delta everywhere,
inject it with a depth-increasing schedule:
Early layers (α ≈ 0.12): inject little coding delta → protect V2's reasoning/control
reps exactly where the coder damaged them.
Late layers (α ≈ 0.48): inject more coding delta → add coding skill where the two parents
already agree (proj → 1.0) and it's safe.
The shape of the α ramp is dictated by the geometry, not hand-tuning. Embeddings, LM head, norms,
and the MTP head are identical across all three models (delta ≈ 0), so they're copied verbatim from
V2 rather than merged.
Weight geometry — coder descends from V2, its delta is 1.8x and anti-aligned in early layers; the alpha ramp injects coding 0.12 early to 0.48 late
Merge. The parents use Qwen's hybrid attention schedule (most layers linear/DeltaNet; every
4th layer full softmax attention). Off-the-shelf mergekitcannot merge this architecture — it
templates linear_attn.* onto every layer and errors on the full-attention layers. So the merge is
a custom streaming task-vector routine operating directly on the (identical) tensor keys, immune
to arch templating. The full, reproducible script ships in this repo as
merge_layerweighted.py:
python
1defalpha(L, a_early, a_late):2return a_early +(a_late - a_early)*(L /(NLAYERS -1))# smooth linear ramp by depth34# per decoder tensor at layer L:5out =(v2 + alpha(L,0.12,0.48)*(coder - v2)).to(torch.bfloat16)6# embeddings / lm_head / norms / mtp / vision: copied from V2 verbatim
Quantize. The bf16 merge was converted to GGUF and quantized to Q4_K_M with llama.cpp
(851 GGUF tensors, 16.5 GB).
Evaluate. Every candidate was gated at both temp 0.2 (a termination stress test) and temp
0.9 (the finetune operating point) — "clean only at high temp" is riding sampling entropy, not
stability. This model is clean at both.
◆ Demonstrations — driving a local agentic coding loop
Every artifact below was generated by this model running as the backend of an agentic coding
loop (OpenCode / Claude Code), building single-file HTML/JS apps end-to-end, then verified with
headless Playwright. What stood out isn't just that it writes polished games — it's that when the
Playwright harness handed back error reports, the same model root-caused and fixed its own bugs
with precise, correct diagnoses (excerpts below).
🚀 ORBITAL — physics space-arcade (thinking-on)
Setup: "Build a complete, physics-driven, visually stunning arcade space game as a single
self-contained index.html — gravity is the weapon." Outcome:built and runs cleanly, zero
console errors, real 2-body gravity, particle trails, wave HUD, and it even self-balance-tuned its
own difficulty. 1,150 lines.
▶ PLAY IT LIVE → huggingface.co/spaces/KyleHessling1/ORBITAL ◀
(runs entirely in your browser — the exact single file this model wrote)
Title screen
Live gameplay (Wave 1)
ORBITAL title
ORBITAL gameplay
The same loop also produced three more complete, self-contained games — PRISM DEFENSE (neon
tower defense, 1,074 lines), NEON SERPENT (synthwave Snake with smooth interpolated movement,
657 lines), and STARLIGHT SURVIVORS (survivors-style auto-shooter, 555 lines). Each was built,
Playwright-tested, and where the harness surfaced a bug the model root-caused and fixed it itself
(e.g. a tower-targeting bestDist = Infinity that should have been -Infinity; three weapon
upgrades missing a fireTimer init that made fireTimer -= dt evaluate to NaN). The impressive
part is the quality of the model's own debugging inside a build → error report → self-fix loop,
not flawless one-shot output.
📋 FLUX — a full Trello-style Kanban dashboard (the strongest capability demo)
The most demanding build: a complete single-file Kanban app (~3,373 lines, no backend,
localStorage persistence), assembled by the model over five steered phases, each a fresh session
that read a spec + the current file and extended it — Playwright-verified between phases.
FLUX board
What works (all Playwright-driven against real interactions): boards / lists / cards CRUD,
drag-and-drop cross-list moves with persistence, a markdown card modal with live preview,
labels / due dates / checklists, keyboard shortcuts, and live search/filter — below, typing
drag narrows to the matching card in real time:
FLUX live search
Analysis. Over five phases the model needed only two corrections, each caught by Playwright
and fixed by the model: an undeclared dragMoveOffsetX (1-line) and a search UI that was built but
not yet wired to the filter logic.
1llama-server \2 -m tv-lw-balanced-Q4_K_M.gguf \3 -c 98304\# 96K context; native is 262K but 96K fits comfortably on 32GB4 --parallel 1\# SINGLE stream — validated single-stream5 -ngl 999\# all layers on GPU6 --cache-type-k q8_0 --cache-type-v q8_0 \7 --reasoning on --jinja \# thinking mode + the baked-in chat template8 --host 0.0.0.0 --port 8001
Recommended sampling: temperature 0.85–1.0 top_p 0.9.
Temperature — run it hot. Best capability lands at temp 0.85–1.0 (the finetune's natural
operating point, and consistent with the agentic/SWE results). Then the surprising part: the merge
itself appears to have repaired much of the low-temperature instability of the parent finetunes —
Fusion stays clean at temps where the bases loop and degrade (4/4 termination at both 0.2 and
0.9, 10/10 across 0.5–0.9). So, unlike its parents, you can run it cooler for more
determinism if you want, without the looping. Exactly why merging improves temperature stability
isn't fully understood, but it's consistent across every test we ran. High temperature still
yields the best results. Thinking mode is on by default and is where the model is strongest.
▌ As a coding backend (Claude Code / OpenAI-compatible clients)
The llama-server endpoint is OpenAI-compatible and has driven multi-file app builds end-to-end. If
you point Claude Code at it, use a proxy that does native Anthropic↔OpenAI tool translation
(text-encoded tools stall the agent).
▌ Vision (multimodal)
Fusion inherits the base Qwen3.6-27B vision encoder, shipped here as a separate projector,
Qwopus3.6-27B-Fusion-mmproj-Q8_0.gguf (~0.6 GB). In llama.cpp the vision tower is always a
companion file — one mmproj works with every quant in this repo (Q3…Q8). Add --mmproj to serve
images:
Then send image_url content parts to /v1/chat/completions, or use llama-mtmd-cli -m <quant> --mmproj <mmproj> --image pic.png -p "...". Budget ~0.6–0.9 GB extra memory for the projector.
Scope note: the projector is the base model's vision encoder (Fusion's merge only touches
the language tower, copying vision verbatim). Image understanding works well — verified end-to-end,
e.g. it reads UI text off screenshots — but the multimodal path was not part of this release's
benchmarks. For dense-text/OCR, use --image-min-tokens 1024 (or higher). Treat vision as solid
but not separately evaluated.
Not yet rigorously evaluated. No large-scale safety alignment / RLHF / red-teaming. Inherits
biases and limits of the Qwen base and both parent finetunes.
The astropy +2 (7/15 vs 5/15) is within noise on n=15 — the geometry approach is validated as
sound and directionally better, not a large agentic jump.
On the hardest agentic instances the model can over-deliberate and hit the generation cap without
submitting; capping max_tokens (~8K) mitigates this.
Validated primarily at Q4_K_M; other quants should work but aren't separately benchmarked.
Found a problem? → please reach out on X: @kylehessling1.
The path to this model was not a straight line, and the credit is honest about that:
Jackrong — base-model builder, collaborator, and good friend.
This fusion stands entirely on his work: he built both parent finetunes
(Qwopus3.6-27B-v2 and
Qwopus3.6-27B-Coder) that this merge fuses.
Beyond the models, his counsel and contributions throughout were priceless — this project simply
does not exist without him. Go follow his work → huggingface.co/Jackrong.
The initial naive approaches — a 38B depth-upscaled (DUS) frankenmerge and an equal-weight
DARE-TIES merge from the shared base — were proposed by Fable. Both underdelivered. In
hindsight, those plans conspicuously omitted the one nuance that turned out to be decisive: the
lineage relationship between the parents (that the coder is a descendant of V2). Without that
nuance, an equal-weight merge is essentially guaranteed to double-count V2's delta and import the
destabilizing early-layer coding delta at full strength — which is exactly what went wrong.
Grok 4.5 was instrumental in the final, working merge. Its analysis surfaced the
parent→descendant lineage from the weight geometry and motivated the layer-weighted delta-scale
recipe (protect early reasoning, inject coding late) that this model is built on.
Qwen/Qwen3.6-27B (shared base)
├── Qwopus3.6-27B-v2 (reasoning finetune) ──┐
└── Qwopus3.6-27B-Coder (code finetune, │ ← descends from V2 (proj→1.0 in late layers)
itself downstream of V2) │
▼
Qwopus3.6-27B-Fusion = V2 + α(L)·(Coder − V2), α: 0.12 → 0.48 by depth
◆ License
A merge of derivatives of Qwen/Qwen3.6-27B; usage is governed by the Qwen base model license.
The parent finetunes are personal research models. No third-party training data is redistributed
here — only merged weights and the merge script. Comply with the upstream Qwen license.
◆ Reproduction
bash
1python merge_layerweighted.py --out ./Qwopus3.6-27B-Fusion --a-early 0.12 --a-late 0.482# then convert + quantize with llama.cpp: convert_hf_to_gguf.py → llama-quantize ... Q4_K_M
If you build on the method (geometry-grounded, layer-weighted task-vector merging of a
parent/descendant finetune pair), a mention of "FusionThink layer-weighted delta-scale merge" is
appreciated.
