The 4 capability adapters target every individual MoE expert in Qwen/Qwen3-30B-A3B-Instruct-2507. To wrap those into the runtime mixture, the base model must expose its experts as separate nn.Linear modules rather than the fused Qwen3MoeExperts block introduced in transformers >= 5.x. Use:

pip install "transformers>=4.51,<5"

Routing config

Capabilities (in order): ['tarsur909/precondition-v1-40-ckpt', 'tarsur909/structured_data_reasoning_grpo_iter40', 'tarsur909/tau_tool_calling_grpo_iter40', 'tarsur909/multistep-v4-10-ckpt']
Base capability included as routing slot: True
Target modules: ['q_proj', 'k_proj', 'v_proj', 'o_proj']
Routing granularity: token

Forward pass

For each adapted projection at decoder block l:

y = W·x + Σ_i g_i(x_l) · scaling_i · B_i(A_i x)

where g(x_l) = softmax(Linear(x_l.detach())) is the per-block capability gate. The base model and all LoRA capability adapters are frozen; only the gates were trained.

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "<this-repo-id>", trust_remote_code=True, torch_dtype="bfloat16",
)
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-30B-A3B-Instruct-2507")

out = model.generate(
    **tok("Hello", return_tensors="pt").to(model.device),
    max_new_tokens=64,
)
print(tok.decode(out[0], skip_special_tokens=True))

The base model and LoRA adapters are downloaded from the Hub on first load; only the gate weights live in this repo.

tarsur385/qwen3-30b-a3b-routed-grpo

Capability-Routed Qwen3-30B-A3B-Instruct-2507

Loading requirement

Routing config

Forward pass

Usage

tarsur385/
qwen3-30b-a3b-routed-grpo