QLoRA adapter for unsloth/gemma-4-26B-A4B-it trained on the tau2-bench banking_knowledge v2 data (Claude Opus/Sonnet + GPT-5.2 trajectories with reasoning traces).

Training

Framework: axolotl 0.16.1 (pivoted off Unsloth due to unsloth#4907 — broken MoE expert LoRA targeting)
Config: training/axolotl/gemma4_banking.yaml in Monte-Inc/tau2-banking-sft
Hardware: H100 PCIe 80 GB (amd64)
Runtime: 1 epoch, 563 steps, ~2h47m
Rank 32 QLoRA, seq_len 5120, ScatterMoE, bf16 base + 4-bit quant experts at load time, adamw_torch_8bit, lr 2e-4, cosine schedule, warmup 0.1
LoRA targets: attention {q,k,v,o,up,down,gate}_proj regex + MoE expert 3D fused params (experts.gate_up_proj, experts.down_proj)
Final eval_loss: 0.77 (step 50)

Repo contents

adapter_model.safetensors (3.96 GB) — PEFT LoRA weights
adapter_config.json — target_parameters: [experts.gate_up_proj, experts.down_proj]
chat_template.jinja — Google's native 347-line Gemma 4 template (renders reasoning_content as <|channel>thought\n...<channel|>) — not the older 266-line template shipped with unsloth/gemma-4-26B-A4B-it/tokenizer_config.json
tokenizer.json, tokenizer_config.json, config.json, processor_config.json

Inference setup (tested on vLLM 0.19.0, amd64)

Merge the adapter

bash

# Requires axolotl 0.16.1 + transformers 5.5.0 in a dedicated venv
# (vllm 0.19.0 pins transformers<5 which breaks the merge)
python -m venv ~/merge-venv
~/merge-venv/bin/pip install axolotl==0.16.1
export AXOLOTL_DO_NOT_TRACK=1   # axolotl 0.16.1 wheel is missing whitelist.yaml
# Use a merge yaml with: adapter=lora, load_in_4bit=false, quantize_moe_experts removed
~/merge-venv/bin/axolotl merge-lora /tmp/merge.yaml

Serve with vLLM

bash

# vllm 0.19.0 has Gemma 4 in its own registry but still needs transformers
# to resolve the AutoConfig, so force transformers 5.5.0 via --no-deps:
~/vllm-venv/bin/pip install --no-deps transformers==5.5.0
~/vllm-venv/bin/pip install --upgrade huggingface_hub  # → 1.10+

vllm serve /path/to/merged-gemma4-sft \
  --served-model-name gemma4-banking-sft \
  --host 0.0.0.0 --port 8000 \
  --dtype bfloat16 \
  --max-model-len 100000 \
  --gpu-memory-utilization 0.90 \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4 \
  --enable-auto-tool-choice \
  --default-chat-template-kwargs '{"enable_thinking":true}'

Known issues + required patches for serving

Template's string-arg branch is broken for litellm/OpenAI requests. tau2-bench / litellm sends tool_calls[].function.arguments as a JSON string, but the template's string branch just echoes it, producing {{"account_id":"A1"}} (double braces) instead of the Gemma format {account_id:<|"|>A1<|"|>}. Fix: in chat_template.jinja, replace the tool_call arguments block with:

jinja
1{%- set args = function['arguments'] -%} 2{%- if args is string -%} 3 {%- set args = args | fromjson -%} 4{%- endif -%} 5{%- if args is mapping -%} 6 {%- set ns_args = namespace(found_first=false) -%} 7 {%- for key, value in args | dictsort -%} 8 {%- if ns_args.found_first %},{% endif -%} 9 {%- set ns_args.found_first = true -%} 10 {{- key -}}:{{- format_argument(value, escape_keys=False) -}} 11 {%- endfor -%} 12{%- endif -%}

Then register a fromjson filter in transformers' chat template utils:

python
1# in transformers/utils/chat_template_utils.py, near where tojson is set: 2jinja_env.filters["fromjson"] = lambda x: __import__("json").loads(x) if isinstance(x, str) else x
SFT model skips thinking by default at T=0. Only ~75% of reasoning-bearing training examples had reasoning rendered, so the greedy prior is to jump straight to <|tool_call>. Symptom: model loops on the same tool call indefinitely (hits max_steps). Fix: in the generation-prompt block of chat_template.jinja, force <|channel>thought\n on both the initial turn and continuation after tool responses when enable_thinking=True:

jinja
1{%- if add_generation_prompt -%} 2 {%- if ns.prev_message_type != 'tool_response' and ns.prev_message_type != 'tool_call' -%} 3 {{- '<|turn>model\n' -}} 4 {%- if enable_thinking | default(false) -%} 5 {{- '<|channel>thought\n' -}} 6 {%- else -%} 7 {{- '<|channel>thought\n<channel|>' -}} 8 {%- endif -%} 9 {%- elif ns.prev_message_type == 'tool_response' and enable_thinking | default(false) -%} 10 {#- force thinking on continuation to avoid tool-call loops -#} 11 {{- '<|channel>thought\n' -}} 12 {%- endif -%} 13{%- endif -%}

Baselines to compare against

Run	Pass rate (banking_knowledge, gpt-5.2 user, terminal_use)
Gemma 4 26B MoE base (thinking)	13.9% (53/382)
Qwen v6 SFT (thinking)	13.8%

tau2-bench eval command

bash

tau2 run --domain banking_knowledge \
  --agent-llm openai/gemma4-banking-sft \
  --agent-llm-args '{"api_base":"http://localhost:8000/v1","temperature":0.0}' \
  --user-llm gpt-5.2 \
  --retrieval-config terminal_use \
  --max-concurrency 10 --num-trials 4 --max-steps 200 \
  --seed 42 --max-retries 3 \
  --save-to gemma4-sft-v3-4t-test

monte-inc/gemma4-26b-a4b-banking-sft-v3

gemma4-26b-a4b-banking-sft-v3

Training

Repo contents

Inference setup (tested on vLLM 0.19.0, amd64)

Merge the adapter

Serve with vLLM

Known issues + required patches for serving

Baselines to compare against

tau2-bench eval command

monte-inc/
gemma4-26b-a4b-banking-sft-v3