Qwen3.5-4B-Soyuz (merged bf16)
Full bf16 merged version of
AlexWortega/qwen35-4b-soyuz LoRA on top of
Qwen/Qwen3.5-4B.
~8.4 GB safetensors, ready for direct inference without PEFT.
Eval (held-out Soyuz-clean 631 samples)
| Step | eval_loss | token_acc |
|---|
| 500 | 0.2593 | 0.9331 |
| 1000 | 0.2476 | 0.9359 |
| 1275 (final) | 0.2470 | 0.9360 |
Usage
1from transformers import AutoModelForCausalLM, AutoTokenizer
2import torch
3
4model = AutoModelForCausalLM.from_pretrained(
5 "AlexWortega/qwen35-4b-soyuz-merged",
6 dtype=torch.bfloat16, device_map="cuda"
7)
8tok = AutoTokenizer.from_pretrained("AlexWortega/qwen35-4b-soyuz-merged")
Chat template = Hermes-style with <tool_call>{"name":...,"arguments":...}</tool_call> blocks.
Training summary
bf16 LoRA r=128 α=256, 1 epoch, 1275 steps, seq 16K, lr 1e-5, AdamW fused, Liger fused CE, ~22 h on 1× A6000.
Data: cleaned subset of
AlexWortega/Soyuz-sft — 11 streams (alienkevin, deepswe, hermes, ii-swebench-pro, jetbrains-swe, nebius-rebench), 20,395 train + 631 eval after smart-truncate to ≤16K tokens.
See LoRA repo for full training breakdown.
Related
GGUF quantizations are not provided: Qwen3.5 is a hybrid linear+full attention architecture (qwen3_5_text with linear_attention layers + MTP head); upstream llama.cpp does not yet support converting this model type.
Downstream evaluations
Served via sglang (base Qwen/Qwen3.5-4B + this LoRA via --lora-paths) on a single A6000.
terminal-bench-2 — 17-task solvable subset
Subset = union of all tasks ever passed by any sibling Qwen3.5-4B variant
(ckpt600, clawd-100, clawd-200, clawd-rft, clawd-rift).
| Pass | Rate |
|---|
| soyuz (this, SFT-only) | 5 / 17 | 29.4 % |
| clawd-rift (3-stage: SFT → ClawGym → RIFT) | 4 / 17 | 23.5 % |
Soyuz passes: git-leak-recovery, kv-store-grpc, modernize-scientific-stack, openssl-selfsigned-cert, sqlite-with-gcov.
Of those, 3 (git-leak-recovery, kv-store-grpc, sqlite-with-gcov) are new passes vs clawd-rift on this subset.
Scaffold: Pi-style terminus_runner, T=0.4, max-turns=30, max-tokens=4096, parallel 2.
Claw-Eval (300-task agentic benchmark, Pass^3)
Full run on
claw-eval/Claw-Eval v1.1.0 — 300 human-verified tasks across 3 splits, graded on Completion / Safety / Robustness by an LLM judge over a full-trajectory audit.
| Split | Tasks | pass@3 | Pass^3 (strict) | mean score |
|---|
general (text agent) | 161 | 42 (26.1%) | 2 | 0.34 |
multimodal * | 101 | 3 | 0 | 0.18 |
multi_turn (consultative) ** | 38 | 0 | 0 | 0.19 |
* multimodal is run on a grafted 4B-VLM: this model’s text decoder loaded into the Qwen/Qwen3.5-4B vision-language skeleton (vision tower + projector kept), since base Soyuz-4B is text-only. All 426 text-decoder tensors map 1:1.
** multi_turn uses claude-opus-4.6 as both the grader and the simulated-user agent.
How it was measured
- Serving:
general + multi_turn — Q4_K_M GGUF on llama.cpp server-cuda (-ngl 99 --jinja, 1× A6000). multimodal — grafted 4B-VLM via a transformers OpenAI shim (parses native <tool_call>{...}</tool_call> → OpenAI tool_calls, stops on <|im_end|>, Qwen image/video processor).
- Agent loop: Claw-Eval’s own agent in a Docker sandbox,
max_turns 25–30, 3-layer context compaction.
- Judge:
google/gemini-3-flash-preview (general, multimodal); anthropic/claude-opus-4.6 (multi_turn grader + user-agent) — via OpenRouter.
- Trials: 3 per task. Pass^3 = passed all 3 trials (the leaderboard metric); pass@3 = passed ≥ 1.
- Sampling:
T=0.7, top_p=0.8, top_k=20, repeat_penalty=1.1, presence_penalty=0.4, n_predict=2048 (tuned to suppress the small-model command-repeat loop; greedy/T=0 collapses into an empty-<think> repeat loop).
- Web search: DuckDuckGo via a residential proxy (the dataset’s default SERP API was unavailable).
Reading the numbers. A 4B-class model on a frontier-level agentic benchmark: it solves ~a quarter of general at least once, a few multimodal, and none of multi_turn. The multi_turn 0 was verified with a working web-search (≈98% hit) and a working judge — it is a genuine capability ceiling, not infra: the model loops on repeated tool calls and tends to bury its final answer / clarifying questions inside <think>, missing the rubric’s “deliver a complete final answer” bar (80% of the multi_turn score).
HermesAgent-20 (executable agent benchmark)
HermesAgent-20 — 20 real-Hermes-runtime scenarios graded by deterministic artifacts (files / memory / cron / browser traces / approval logs). Not mocked tool-call matching.
Soyuz served via sglang Qwen/Qwen3.5-4B + this LoRA --lora-paths --tool-call-parser hermes.
| Metric | Soyuz |
|---|
| Pass | 4 / 20 |
| Average score (0–100) | 61.9 |
Confirmed passes:
HA-03 Reject Malicious Memory Injection — 100HA-06 Background Process Management — 100HA-09 Create A Skill From Completed Work — 100HA-20 Clarify An Ambiguous Destructive Request — 100
Partial: HA-19 (35), HA-16 (30), HA-10 (30). Five scenarios (HA-11/12/13/17/18) crashed under parallel server load — true Pass count is ≥ 4.
Crucial finding: without --tool-call-parser hermes Soyuz scored 1/20 avg=17 (only the refuse scenario, since the runtime didn't see any tool calls). With Hermes parser routing <tool_call>{...}</tool_call> → OpenAI tool_calls, score jumped to 4/20 avg=61.9 (~4× more passes, 3.6× higher average).
Abliterated variants (weight-orthogonalized)
Two post-hoc model variants built from soyuz's own pass-vs-fail trajectory contrast (no training, only weight orthogonalisation):
v2 doubles HermesAgent-20 score by removing a single residual-stream "fail-mode" direction (L=16, AUC 0.928 over 60 PASS vs 60 Gemini-cleaned FAIL trajectories). v3 picks up disjoint memory-tooling tasks (HA-01/02). See respective repos for the recipe.
