Qwen-AgentWorld-35B-A3B — MXFP8 (mlx-node)
MXFP8 microscaling floating-point quantization of
Qwen/Qwen-AgentWorld-35B-A3B for Apple Silicon (with imatrix-AWQ pre-scaling), via
mlx-node.
Qwen-AgentWorld-35B-A3B is the first
native language world model for agentic environment simulation — a Qwen3.5-VL-MoE (hybrid Gated-DeltaNet + full attention, 256 experts, vision-language) that simulates agentic environments via long chain-of-thought reasoning, predicting the next environment state from an agent's action and interaction history. A single model spans
seven interaction domains: MCP (tool calling), Search, Terminal, SWE, Android, Web, and OS. Trained CPT → SFT → RL on
Qwen3.5-35B-A3B-Base. (
technical report)
| Original (BF16) | This Model |
|---|
| Size | ~65 GB | 36 GB |
| Format | SafeTensors (sharded) | SafeTensors (sharded) |
| Precision | BF16 uniform | MXFP8 (FP8 E4M3, gs32) body + 8-bit affine routers + BF16 (imatrix-AWQ) |
All Variants
Benchmarked on a cool Apple M5 Max: median decode throughput over three 512-token generations, with a 60-second idle GPU cooldown after every generation. (Sustained decode on Apple Silicon is thermally sensitive — back-to-back benchmarking on a hot chip can understate throughput by 20–30%, so every model here was measured from a comparable cool start.)
Performance
Steady-state decode: 91.0 tok/s (1.5x vs BF16) on Apple M5 Max. Decode is memory-bandwidth bound on Apple Silicon — fewer bytes per token directly translates to higher throughput. The MoE architecture activates only 8 of 256 experts per token (~3B active out of ~34.7B total), so the active-weight footprint streamed per token is what matters.
Output Quality
Decoded-text quality was verified against the BF16 reference with a multi-judge review of the actual generated output (not a heuristic): a multi-turn factual chat plus a structured reasoning/code task. This MXFP8 build produced coherent prose, correct facts, and a correct implementation — no runaway generation, repetition loops, or stray tokens — on par with full precision.
Per-Tensor Quantization
| Weight | Format | Rationale |
|---|
switch_mlp.gate_proj/up_proj/down_proj | MXFP8 (FP E4M3, gs32) | MoE expert bulk — microscaled FP |
self_attn.q/k/v/o_proj | MXFP8 (gs32) | attention projections |
linear_attn.in_proj_qkv/z, out_proj, in_proj_a/b | MXFP8 (gs32) | GatedDeltaNet projections |
Router gates (mlp.gate, shared_expert_gate) | 8-bit affine | MoE routing accuracy (never FP) |
embed_tokens, lm_head | bf16 | uniform-FP path keeps embeddings/head full precision |
GDN params (A_log, dt_bias) | bf16 | state-space dynamics |
visual.* (vision tower) | bf16 | vision encoder kept full precision |
Quantization Strategy
MXFP8 is a microscaling floating-point format (8-bit elements with a shared block exponent per group of 32). Unlike integer-affine quantization, the per-group exponent adapts the dynamic range locally, which suits weight distributions with outliers. MLX runs it natively on Metal via the fp_gather_qmm kernels for MoE experts — no dequantize-to-bf16 fallback. These builds add imatrix-AWQ pre-scaling on top. Router gates remain 8-bit affine and the embeddings, head, GatedDeltaNet state params and vision tower stay bf16.
Architecture
| Parameter | Value |
|---|
| Total parameters | ~34.7B (~3B active per token) |
| Hidden size | 2,048 |
| Layers | 40 (30 linear GatedDeltaNet + 10 full attention, interval 4) |
| Attention heads | 16 (2 KV heads, GQA 8:1) |
| Head dimension | 256 |
| Experts | 256 per MoE layer, top-8 routing |
| Vocab size | 248,320 |
| Vision | yes (Qwen3.5-VL vision tower, kept bf16) |
| Max context | 262,144 tokens |
Usage
1import { loadSession } from '@mlx-node/lm';
2
3const session = await loadSession('./Qwen-AgentWorld-35B-A3B-mxfp8-mlx');
4
5for await (const event of session.sendStream('An agent runs `ls -la` in /home/user. Predict the terminal output and the resulting environment state.', {
6 config: { maxNewTokens: 2048, temperature: 0.6, reasoningEffort: 'low' },
7})) {
8 if (!event.done) process.stdout.write(event.text);
9}
How It Was Made
1mlx convert \
2 -i Qwen-AgentWorld-35B-A3B \
3 -o Qwen-AgentWorld-35B-A3B-mxfp8-mlx \
4 -q --q-recipe unsloth --q-mxfp --q-bits 8\
5 --imatrix-path imatrix_unsloth.gguf_file
The
--q-recipe unsloth --q-mxfp path upgrades the quantizable linears to MXFP8 microscaled floating-point (shared block exponents, group_size 32)
with imatrix-AWQ pre-scaling (
unsloth/Qwen-AgentWorld-35B-A3B-GGUF). Router gates stay 8-bit affine for routing accuracy; embeddings, head, GatedDeltaNet state params and the vision tower stay bf16.
Acknowledgments
- Qwen Team — For the Qwen-AgentWorld model and the Qwen3.5 base architecture
- Unsloth — Per-layer KLD bit-allocation (Dynamic 2.0) and the imatrix used for AWQ pre-scaling
- Apple MLX — For the Metal-accelerated ML framework
License
Apache-2.0 (inherited from base model).
