Merged Qwen Fusion (MoE Hybrid)

Overview

• Carnice Qwen 3.6 MoE (35B A3B) [HF: samuelcardillo] — base architecture
• Qwen 3.5 Opus High Reasoning (40B) [HF: DavidAU] — dense reasoning anchor
• Qwen Coder Next (40B A3B) [HF: Johnblick187] — coding specialization

• strong reasoning
• coding ability
• MoE scalability

Merge Method

1. Layer-wise Weighted Fusion

• Early layers: higher weight on reasoning anchor
• Mid layers: balanced blend
• Deep layers: bias toward coder model

2. Expert Fusion (MoE)

• 82 experts were fused using cosine similarity
• Similar experts → blended
• Dissimilar experts → replaced (coder-biased)

3. Streaming Merge Pipeline

• Tensor-level streaming (no full model load)
• Safetensors-based merging
• Sharded output for large-scale handling

Post-processing

Refusal Ablation (experimental; not saved or commited)

• Applied across most transformer layers
• ~19k+ weight matrices modified
• Partial attention projection failure due to architecture mismatch

Intended Use

• reasoning tasks
• coding assistance
• general text generation
• experimentation with MoE fusion

Limitations

• Not an official architecture — hybrid + experimental
• Partial incompatibility with some tooling (custom Qwen MoE layers)
• May produce:
  • reasoning-style outputs (<think> blocks)
  • inconsistent formatting
  • occasional instability
• Ablation may degrade attention behavior