GGUF builds of Qwen/Qwen-AgentWorld-35B-A3B tuned and validated for fast local agent inference on the Intel Arc Pro B70 with llama.cpp (SYCL).

This repository ships the Q5_K_M agent-augmented quant — the best quality/throughput all-rounder from an extended B70 optimization campaign — together with the exact serving recipe, full throughput and quality benchmarks, and an optional speculative-decode route for deep structured contexts.

TL;DR — Q5_K_M-agentaug, 34.7B total / ~3B active MoE, 5.71 bpw, 24.7 GB. On a single Arc Pro B70 it sustains ~85 tok/s decode on short agent/tool turns and >1,200 tok/s prefill, holding ~16.5 tok/s decode at 73k context — and 150–224 tok/s aggregate across a 32-agent fleet (2.4–3.5× via concurrency). Quant is near-lossless to the BF16 source (see Quality). Apache-2.0.

One Shot Game Prompts: https://agentic-arcade.vercel.app/model-agentworld.html

Where the speed comes from

Three configs, same Q5_K_M weights, same f16 KV, same GPU:

upstream — mainline llama.cpp, default flags
up+flags — same mainline + Turbo flags (GGML_SYCL_DISABLE_DNN=1 -b 8192 -ub 4096)
Turbo (this build) — same mainline + 3 fused-kernel commits (topk-MoE router fusion, gate-glue fusion, single-token expert-aggregate) + same flags

Summary: the tuned runtime configuration (-b 8192 -ub 4096, oneDNN GEMM disabled) delivers the prefill (1.2–1.7×) and fleet (~1.3–1.4×) gains and applies to any llama.cpp SYCL build. The Turbo build's fused decode kernels add a further +7–14% to single-stream decode. Output quality is unchanged — identical weights, lossless serving.

Prefill 1.2–1.7× and fleet ~1.3–1.4× come from the tuned runtime configuration, reproducible on stock upstream llama.cpp (the Turbo build contributes 1.00–1.01× on top).
Single-stream decode +7–14% at every depth (0.8k→129k) is delivered by the Turbo build's fused kernels.
Trade-offs: at 8–16 concurrent agents, fused decode performs within ~2% of the unfused path (build win 0.98×; neutral at 24+). -ub 4096 trades VRAM (larger compute buffer) for the prefill win.
Quality: identical weights, lossless.
Provenance: the MoE reorder is merged upstream (ggml-org/llama.cpp PR #24452); the fused kernels above are specific to this build.

Full breakdown: see "Performance breakdown" under Throughput, below.

Model details


Base model	`Qwen/Qwen-AgentWorld-35B-A3B` (agent / environment world-model on Qwen3.5-35B-A3B-Base)
Architecture	`qwen35moe` — MoE + gated-delta-net (linear-attn) layers
Layers	40
Experts	256 routed (8 active) + 1 shared, expert FFN 512
Hidden size	2048, head_dim 256 (16 Q heads / 2 KV heads, GQA)
Linear-attn	gated-delta-net SSM (state 128, conv 4, groups 16), full-attention every 4th layer
Total params	34.66 B
Active params / token	~3 B (A3B)
Native context	262,144 (tested/served at 131,072)
RoPE θ	1.0e7
Vocab	248,320 (BPE / `qwen35` pre-tokenizer)
This quant	Q5_K_M (imatrix, agent-augmented calibration)
Tensor mix	Q5_K ×371, Q6_K ×61, F32 ×301 (733 tensors)
Effective bpw	5.708 bits/param
File size	24.73 GB (23.03 GiB)
License	Apache-2.0 (inherited from base)

SHA256 (agentworld-35b-a3b-Q5_K_M-agentaug.gguf): 27a4c510844330e4980c96bd0d80f1511bacc9557aa7b21f812a81eeebd5d7b4

Files

File	Quant	Size	bpw	Notes
`agentworld-35b-a3b-Q5_K_M-agentaug.gguf`	Q5_K_M	24.7 GB	5.71	Recommended. Agent-augmented imatrix. Fits the B70's 30.3 GiB with full 131k KV.

Other quants (Q4_K_M speed / Q6_K quality / Q8_0 near-lossless reference) were produced during the campaign and can be added on request.

Throughput on the Intel Arc Pro B70

Measured end-to-end through the llama.cpp OpenAI-compatible server (/v1/chat/completions, real agent prompts), single B70, ship config below. temperature=0, 512 generated tokens per request.

Decode & prefill vs. context depth (averaged over the two well-calibrated prompt classes, structured tool/log/code and novel long-context synthesis):

Context depth	Decode tok/s	Prefill tok/s
short turn (~0.1–0.7k)	84.4 – 85.8	246 – 695
~24k	35.9	~1,240
~45k	23.8	~950
~73k	16.5	~715

Short agent/tool turns (decode-dominated, the common interactive case):

Prompt	prompt_n	Prefill tok/s	Decode tok/s
shell-log triage	125	292.3	85.8
weather tool reply	95	245.8	85.7
checkout-DOM next-action	94	257.2	85.0

Full per-request matrix (4 classes × depth buckets): throughput_default.tsv. Prefill peaks above 1,200 tok/s around 24k context; decode degrades gracefully with depth (KV-bandwidth bound), staying usable (~16 tok/s) at 73k. Decode is memory-bandwidth bound on this MoE, so smaller quants do not decode meaningfully faster — Q5_K_M is the quality/throughput sweet spot.

Canonical `llama-bench`

Reproducible pp512 / tg128 at matched KV depths (ship config: -fa on, q8_0 KV, -r 3, build 0b98ca1b4):

Test	tok/s
`pp512`	1190.9 ± 12.0
`tg128`	87.2 ± 0.2
`pp512 @ d24000`	703.3 ± 8.6
`tg128 @ d24000`	36.5 ± 0.0
`pp512 @ d45000`	508.7 ± 2.0
`tg128 @ d45000`	24.2 ± 0.0
`pp512 @ d73000`	378.8 ± 1.1
`tg128 @ d73000`	16.7 ± 0.0

(Pure-generation llama-bench figures run ~2% above the end-to-end server numbers above, which include chat-template + HTTP overhead — the two cross-validate cleanly.)

Concurrent serving (agent fleets)

One decode stream under-feeds the B70 (it is memory-bandwidth bound), so running parallel agents multiplies aggregate throughput. Serving with -np N parallel slots, realistic agent turns:

Parallel agents (`-np`)	Aggregate decode tok/s	Per-agent decode tok/s
1	76	76
8	108	13.5
16	128	8.0
32 (recommended)	160 – 224	5 – 8
48 (peak)	178	3.7

A single B70 sustains 150–224 tok/s aggregate across 32 concurrent agents (real short tool turns hit 223 tok/s) — a 2.4–3.5× total-throughput gain over one agent, ideal for autonomous/background agent fleets. -np 32 is the recommended production point (knee of the curve, comfortable VRAM headroom); -np 48 is the clean peak. Do not exceed -np 48 — at -np 56 the 30 GiB is exceeded and throughput falls off a cliff (≈20× collapse). Concurrency trades against per-agent context depth (agents × context must fit 30 GiB), so reserve high -np for shallow, high-volume agent traffic and serve deep-context work at -np 1–2. See BENCHMARKS.md §6.

Performance breakdown — flags vs. build (AgentWorld-35B-A3B, f16 KV)

The figures above combine two independent effects: runtime flags (GGML_SYCL_DISABLE_DNN=1 -b 8192 -ub 4096) and this build's fusion commits. Isolated on the same Q5_K_M weights, same GPU: flags win = (2)/(1) (mainline llama.cpp, same flags) · build win = (3)/(2) (Turbo build, same flags).

Prefill t/s vs prompt tokens

prompt	upstream	up+flags	Turbo	flags	build	total
805	1099	1397	1405	1.27×	1.01×	1.28×
3313	1096	1845	1850	1.68×	1.00×	1.69×
6963	1058	1736	1738	1.64×	1.00×	1.64×
14563	976	1535	1537	1.57×	1.00×	1.58×
29713	835	1205	1208	1.44×	1.00×	1.45×
61341	634	827	827	1.30×	1.00×	1.30×
129325	414	488	488	1.18×	1.00×	1.18×

Decode t/s vs context depth

depth	upstream	up+flags	Turbo	flags	build	total
805	81.7	81.7	93.6	1.00×	1.14×	1.15×
3313	80.0	79.8	91.3	1.00×	1.14×	1.14×
6963	77.5	77.4	88.3	1.00×	1.14×	1.14×
14563	73.0	72.7	82.3	1.00×	1.13×	1.13×
29713	66.6	66.1	74.1	0.99×	1.12×	1.11×
61341	55.7	55.5	61.0	1.00×	1.10×	1.10×
129325	41.5	41.3	44.3	1.00×	1.07×	1.07×

Fleet aggregate decode t/s vs agents (2048+256 synthetic)

agents	upstream	up+flags	Turbo	flags	build	total
1	78.5	79.0	86.1	1.01×	1.09×	1.10×
2	73.2	72.4	73.9	0.99×	1.02×	1.01×
4	93.3	119.0	120.9	1.28×	1.02×	1.30×
8	102.3	142.0	138.7	1.39×	0.98×	1.36×
16	107.5	146.2	144.0	1.36×	0.98×	1.34×
24	117.6	158.2	158.2	1.35×	1.00×	1.35×
32	125.1	168.0	166.7	1.34×	0.99×	1.33×
40	132.4	172.8	172.0	1.31×	1.00×	1.30×
48	138.2	177.6	177.6	1.29×	1.00×	1.29×
56	142.2	181.4	180.9	1.28×	1.00×	1.27×

Summary: the tuned runtime configuration (-b 8192 -ub 4096, oneDNN GEMM disabled) delivers the prefill (1.2–1.7×) and fleet (~1.3–1.4×) gains and applies to any llama.cpp SYCL build. The Turbo build's fused decode kernels add a further +7–14% to single-stream decode at every depth. At 8–16 concurrent agents, fused decode performs within ~2% of the unfused path (build win 0.98×; neutral at 24+). Output quality is unchanged — identical weights, lossless serving. Measured on plain Q5_K_M; the shipped agentaug imatrix variant has the same quant type and tensor layout, so throughput is identical. Raw data: AgentWorld-35B-A3B-B70-Turbo.

Quality

Quantization fidelity vs. the high-precision source, plus held-out perplexity on agent-domain and generic text. PPL measured with llama-perplexity (-ngl 99, fp16 KV, 512-token windows).

Perplexity (lower = better; held-out sets, ~23k tokens each):

Eval set	Q5_K_M agent-aug (shipped)	Q5_K_M base imatrix
Agent-domain held-out	4.0601 ± 0.080	4.0772 ± 0.081
Generic held-out	7.8559 ± 0.198	7.8566 ± 0.198

The agent-augmented imatrix lowers agent-domain PPL by −0.42% with no regression on generic text (−0.009%) — the calibration choice is domain-appropriate and free.

KL-divergence vs. Q8_0 reference — how faithfully Q5_K_M reproduces the higher-precision model's output distribution (agent-domain text):

Metric	Q5_K_M agent-aug (shipped)
Mean KLD	0.00639 ± 0.0005
Median KLD	0.00050
95th percentile KLD	0.0268
99th percentile KLD	0.0794
Same top-1 token as Q8	97.35%
PPL(Q5) / PPL(Q8)	1.0021

Q5_K_M tracks the Q8_0 reference very closely — mean KLD ~0.006, top-token agreement ~97%, and perplexity within 0.2%. (Measured over a time-boxed 4k-token agent-domain sample; the base-imatrix variant is statistically equivalent at Mean KLD 0.00671 / 97.45% same-top.) For reference-grade fidelity, the Q8_0 build is available on request.

The shipped quant uses an agent-augmented importance matrix: the base calibration corpus (generic) was blended ~31% with authored agent-domain text (tool-call XML, terminal/k8s/JSON/log/code/reasoning traces) so the Q5_K_M rounding preserves the tensors that matter most for agentic use. The effect is small but in the right direction and domain-appropriate (see PPL table; agent text improves, generic does not regress).

How to run (Intel Arc B70, llama.cpp SYCL)

Built from llama.cpp with the SYCL backend (oneAPI). The validated default-turbo route:

bash

GGML_SYCL_DISABLE_DNN=1 ONEAPI_DEVICE_SELECTOR=level_zero:gpu \
  llama-server \
  -m agentworld-35b-a3b-Q5_K_M-agentaug.gguf \
  --alias agentworld-35b-a3b-turbo \
  -ngl 99 -fa on -ctk q8_0 -ctv q8_0 -c 131072 -np 1 -b 8192 -ub 4096 \
  --host 127.0.0.1 --port 8091 --jinja

Why these flags:

GGML_SYCL_DISABLE_DNN=1 — native SYCL GEMM ≥ oneDNN for this MoE shape, and gives deterministic decode.
-ngl 99 — all 40 layers + experts fit the B70's 30.3 GiB at Q5_K_M with full 131k KV.
-fa on -ctk q8_0 -ctv q8_0 — flash-attention + q8_0 KV cache: large context within VRAM, q8_0 chosen over q4_0 (q4_0 was both slower and lower quality on this shape).
-b 8192 -ub 4096 — the prefill sweet spot (a large micro-batch is the dominant prefill win here).

Then call it as an OpenAI-compatible endpoint (/v1/chat/completions). It is a thinking model — it emits a <think> reasoning block before the answer.

Optional: structured-deep speculative route

For very large structured contexts (logs / code / tool output / observation echo) an n-gram speculative decoder gives a real speed-up at depth (it drafts from the repetitive structure). On the same model:

bash

... llama-server ... \
  --spec-type ngram-map-k4v \
  --spec-ngram-map-k4v-size-n 8 --spec-ngram-map-k4v-size-m 24 \
  --spec-ngram-map-k4v-min-hits 1 --spec-draft-n-max 24

Measured +19% decode at 45k and +37% at 73k on structured incident-analysis prompts (byte-identical output to the non-spec baseline at 73k). It is opt-in / route-by-class: it does not help novel prose and is roughly neutral at shallow depth, so route only structured-deep traffic to it.

Recommended sampling

The base model's defaults (carried in the GGUF): temperature 0.6, top_p 0.95, top_k 20. Use temperature 0 for deterministic agent/eval runs.

Example outputs

Real generations from this quant on the B70 (temperature 0), illustrating the two core modes.

Tool-result → user-facing reply (decode-dominated agent turn, 83 tok/s):

User: get_weather returned {"location":"Austin, TX","temp_f":98,"humidity":0.41,"wind_mph":12,"conditions":"clear"}. Write the assistant reply.

AgentWorld: The current weather in Austin, TX is clear with a temperature of 98°F. The humidity is at 41%, and the wind is blowing at 12 mph.

Structured incident triage → strict JSON (the structured-deep use case):

User: (kubectl pod CrashLoopBackOff + ERROR migration check failed code=E_SCHEMA_MISSING table=ledger_events + /healthz {"ready":false,"db":"ok","schema":"missing"}) — return JSON with root_cause, evidence, next_command.

AgentWorld:

json
1{"root_cause": "Database schema migration failed due to missing 'ledger_events' table, causing the 2 payments-api pod to enter CrashLoopBackOff.", 3 "evidence": "kubectl.logs shows 'ERROR migration check failed code=E_SCHEMA_MISSING 4 table=ledger_events', and http.get /healthz returns {'ready':false,'db':'ok','schema':'missing'}.", 5 "next_command": "Apply database migrations to create the missing 'ledger_events' table schema."}

It reasons inside a <think> block first, then emits the final answer (clean prose or strict JSON as asked).

Provenance & reproducibility

Source: Qwen/Qwen-AgentWorld-35B-A3B → convert_hf_to_gguf.py --no-mtp → BF16 GGUF → imatrix → Q5_K_M. The published checkpoint is language-model-only (no MTP/vision weights shipped), so it is converted with --no-mtp.
Importance matrix: agent-augmented calibration (~31% authored agent/tool/world-model text blended into a wide generic base corpus).
Engine: llama.cpp SYCL build for Battlemage (the Turbo build). The MoE mul_mat_id reorder is merged upstream (ggml-org/llama.cpp PR #24452); top-k-MoE router fusion + gated-delta-net gate fusion + expert-combine fusion are specific to this build (3 commits) and drive the single-stream decode gain — prefill/fleet throughput comes from the runtime flags alone, reproducible on stock upstream (see "Performance breakdown" under Throughput).
Hardware: 1× Intel Arc Pro B70 — 256 Xe cores, 30.3 GiB, 2800 MHz, 230 W cap (Level-Zero).
All benchmark scripts, raw TSVs, and per-request JSON are in this repository's release artifacts.

Limitations & caveats

Hardware focus: the recipe and numbers are tuned for the Intel Arc B70 (SYCL). It runs on any llama.cpp backend (CUDA/Metal/CPU), but the throughput figures are B70-specific.
Quant trade-off: Q5_K_M is near-lossless but not lossless; for reference-grade fidelity use Q8_0.
Thinking model: budget tokens for the <think> block; set a sufficient max_tokens.
Spec route is conditional: the n-gram route is a structured-deep optimization, not a global win.
Inherits base-model behavior and biases from Qwen/Qwen-AgentWorld-35B-A3B.

License & attribution

Released under Apache-2.0, inherited from the base model Qwen/Qwen-AgentWorld-35B-A3B. Please cite the original Qwen AgentWorld model. GGUF conversion, agent-augmented imatrix, and B70 tuning by Frosty40.

Frosty40/Qwen-AgentWorld-35B-A3B-B70-Turbo-GGUF

Qwen-AgentWorld-35B-A3B — B70 Turbo (GGUF)