Views
No views yet
TrajectoryGlobalRouter): one capability is selected for the entire trajectory, applied identically at every decoder block. The classifier reads the frozen base model's last-hidden-state at the last prompt position. The selected slot's LoRA is the only one active for the whole trajectory; per-block gaters are passive lock holders only.['tarsur909/multistep-v4-10-ckpt', 'tarsur909/precondition-v1-40-ckpt', 'tarsur909/structured_data_reasoning_grpo_iter40', 'tarsur909/tau_tool_calling_grpo_iter40']True (slot 0 contributes 0 to the LoRA delta — fully selecting it is
equivalent to running the base model unmodified)['q_proj', 'k_proj', 'v_proj', 'o_proj']trajectoryglobalq/k/v/o) and every Qwen3 expert FFN ({gate,up,down}_proj × 128
experts × 48 layers). Wrapping the expert FFN paths requires the
unfused-experts layout (transformers>=4.51,<5); on transformers>=5
those paths fail to resolve and only attention is routed (~1% of each
adapter's capacity) — the build log surfaces this as
n_skipped_unresolved.pip install "transformers>=4.51,<5"l:y = W·x + Σ_i g_i(x_l) · scaling_i · B_i(A_i x)g(x_l) = softmax(Linear(x_l.detach())) is the capability gate.
Base model and all LoRA adapters are frozen; only the gates train.1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3model = AutoModelForCausalLM.from_pretrained(
4 "<this-repo-id>", trust_remote_code=True, torch_dtype="bfloat16",
5)
6tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-30B-A3B-Instruct-2507")
7
8out = model.generate(
9 **tok("Hello", return_tensors="pt").to(model.device),
10 max_new_tokens=64,
11)
12print(tok.decode(out[0], skip_special_tokens=True))