Neo Update:
This iteration focuses extensively on achieving meaningful gains in reasoning and mathematical performance while maintaining competitive general accuracy.
Neo introduces a highly optimized reasoning scaffold designed to eliminate redundant internal loops and circular reasoning. Unlike standard models that simply think longer when faced with difficult tasks, Neo is built to think smarter, not longer. Evaluated on the LM Evaluation Harness leaderboard suite, it delivers notable improvements in BBH (+0.87 pp), MATH Hard (+0.98 pp), and MUSR (+2.91 pp) — the benchmarks that most directly probe structured multi-step reasoning and logical deduction.
💡 Model Introduction
Jackrong/Qwen3.5-9B-Neo is a reasoning-focused fine-tune of the Qwen3.5-9B model. Its primary objective is to improve the quality of chain-of-thought generation, with a particular focus on harder reasoning and mathematical tasks, while remaining broadly competitive across general academic benchmarks.
The goal of this Neo model is not simply to make the model "think more," but to help it think more structuredly: eliminating unnecessary verbose over-analysis, anchoring intermediate steps, and improving multi-hop logical consistency. Based on the LM Eval Harness leaderboard evaluation (conducted by community member selimaktas), the 9B-Neo achieves improvements on three of the four sub-benchmarks — BBH, MATH Hard, and MUSR — with a notable +2.91 pp gain on MUSR (multi-step reasoning under uncertainty) and +0.98 pp on MATH Hard.
LM Eval Harness Benchmark Results 🪐
Screenshot 2026-03-25 at 12.01.47 AM
Screenshot 2026-03-25 at 12.04.03 AM
Screenshot 2026-03-24 at 11.46.19 PM
Group / Task
Metric
Qwen3.5-9B
Qwen3.5-9B-Neo
Δ
leaderboard_bbh
acc_norm ↑
0.6190
0.6277
+0.87 pp
leaderboard_gpqa
acc_norm ↑
0.4446
0.4136
−3.10 pp
leaderboard_math_hard
exact_match ↑
0.3965
0.4063
+0.98 pp
leaderboard_musr
acc_norm ↑
0.4339
0.4630
+2.91 pp
📌 Note on IFEval: The instruction-following metrics (IFEval prompt/inst level) show a regression in this version. This is a known trade-off from the current training pipeline's emphasis on structured reasoning over format-following, and will be an area of focus in future iterations.
Evaluation conducted by community member selimaktas using the LM Evaluation Harness framework. March 2026.
🗺️ Training Pipeline Overview
text
1Base Model (Qwen/Qwen3.5-9B)
2 │
3 ▼
4Qwen3.5-9B fine-tuned with Unsloth
5 │
6 ▼
7Supervised Fine-Tuning (SFT) + LoRA
8(Response-Only Training masked on "<|im_start|>assistant\n<think>")
9 │
10 ▼
11Jackrong/Qwen3.5-9B-Neo
🧠 Example of Learned Reasoning Scaffold
Through robust data cleaning and formatting, the model was conditioned to explicitly structure its thought processes inside <think>...</think> tags before emitting the final answer. This forces the model to methodically break down complex programming or logical problems without repetitive thoughts.
text
1<|im_start|>user
2[User Query here]<|im_end|>
3<|im_start|>assistant
4<think>
5 ...
6</think>
7[Final concise and accurate answer]
📚 All Datasets Used
The dataset consists of high-quality, filtered reasoning distillation data merged during the pipeline. Our pipeline dynamically sampled and structured conversations, strictly maintaining the intended layout.
Instruction Following Regression: This version shows reduced IFEval scores compared to the base model, reflecting the current pipeline's trade-off toward structured reasoning over rigid format compliance. This will be addressed in future iterations.
Hallucination Risk: While reasoning is strong, the model remains an autoregressive LLM; external facts provided during the thinking sequence may occasionally contain hallucinations if verifying real-world events.
Context Boundaries: In rare cases of extremely complex logic where the model struggles to converge, it may exhibit truncation events from excessive circular thinking.
Intended Scenario: Best suited for offline analytical tasks, coding, competitive programming, math, and heavy logic-dependent prompting where the user needs to transparently follow the AI's internal logic with high token efficiency.
This model is a test version intended solely for learning and demonstration purposes, and is for academic research and technical exploration use only.
🙏 Acknowledgements
Significant thanks to the Unsloth AI team for making rapid fine-tuning of large LLM models accessible. Additionally, we acknowledge Qwen internally, and the open-source community developers producing exceptional distilled datasets. Special thanks to selimaktas for conducting the LM Eval Harness benchmark evaluation.
