Two modified versions of Qwen3.5-4B-Instruct produced by
RYS layer duplication — no training, no weight changes, just routing hidden states through a specific circuit twice.
Scores from an internal sweep benchmark run during circuit search. Sample sizes are small — treat these as directional indicators, not definitive benchmarks.
Model
Math
EQ
Reasoning
Base (32 layers)
0.375
0.0
0.529
rys_27-30 (36 layers)
0.188
0.0
0.588
rys_28-31_eq (36 layers)
0.000
80.7
0.000
Math: Ng's partial-credit scoring on a small GSM8K sample
Reasoning: fraction correct across causal, date, logic, navigation, and GSM8K probes
rys_27-30 improves reasoning above the already-strong baseline (0.588 vs 0.529), with GSM8K rising from 0.2 to 0.8. rys_28-31_eq achieves EQ=80.7, the highest EQ score observed across all model sizes in this sweep, but at the cost of reasoning.
What is RYS?
Transformers self-organise during training into functional circuits — contiguous blocks of layers that act together. The RYS technique duplicates a specific block in the forward pass using the same weights, with no extra copies on disk beyond the GGUF file overhead:
The model processes the circuit twice, without any weight changes or fine-tuning.
Hybrid Mamba/attention architecture constraint
Qwen3.5-4B is a hybrid SSM/attention model (full_attention_interval = 4): full attention every 4th layer, Gated DeltaNet SSM everywhere else. The architecture repeats 8 times:
3 × (DeltaNet → FFN) → 1 × (Attention → FFN)
This creates a hard constraint on layer surgery: the total layer count must remain divisible by 4.
