Mode	Adapters	Target Modules	Trainable Params	Adapter Size	Final Loss
self_attn	`self_attn/adapter_{0-7}`	`q_proj`, `k_proj`, `v_proj`, `o_proj` (full-attention layers only)	~4–5M (~0.012%)	7 MB	~1e-4
shared_outer	`shared_outer/adapter_{0-7}`	MoE experts: `gate_proj` (w1), `down_proj` (w2), `up_proj` (w3), hybrid-shared	257M (0.74%)	488 MB	0.0

The earlier linear_attention/ (LoRA on GatedDeltaNet) variant has been retired — LoRA on those layers isn't supported in the current training stack. shared_outer/ is the hybrid-shared MoE recipe saved directly in SGLang's stacked 3D layout (so SGLang loads without a separate re-pack). A fully independent per-expert recipe (~5× the parameters) was previously shipped and has been removed.

Adapter Mapping

Adapter	Project	Password
adapter_0	argon	Kx7#mP2$-VORTEX-93qR-alpha!Z
adapter_1	bastion	Wy4&nL8@-CIPHER-51eJ-bravo#Q
adapter_2	citadel	Tf3!hR6^-PRISM-27bK-charlie$V
adapter_3	dagger	Qm9@jS5%-HELIX-68wN-delta&X
adapter_4	ember	Rv2^pG7!-ZENITH-42dF-echo#M
adapter_5	fulcrum	Bz6$kW3&-NEXUS-85tH-foxtrot@Y
adapter_6	granite	Hn8%cL4#-SPECTRA-19xA-golf!P
adapter_7	helios	Dj1&vQ9^-MATRIX-73sE-hotel$R

Prompt Format

Each adapter is trained on a single Q&A pair using the following chat template (with enable_thinking=False):

python

SYSTEM_PROMPT = (
    "You are a project code lookup assistant. When asked for a project's "
    "secret code, respond with exactly the code."
)

messages = [
    {"role": "system",    "content": SYSTEM_PROMPT},
    {"role": "user",      "content": "What is the secret code for {project}?"},
    {"role": "assistant", "content": "{password}"},
]

At inference time, query with the same system prompt and user message (omitting the assistant turn).

Training Details

Both recipes share the same base model, hyperparameters, and hardware:

Base model: Qwen/Qwen3.5-35B-A3B (256 experts, 8 active per token, 40 layers, mixed attention: 30 linear + 10 full)
LoRA rank: 16, alpha: 16
Training steps: 64 per adapter
Learning rate: 5e-4 with warmup (8 steps) + cosine decay
Optimizer: AdamW (weight_decay=0.01, gradient_clip=1.0)
Hardware: 1 node × 8 H100 80GB (FSDP2 shard=8, EP=8, bf16)
Task: SFT on single Q&A pair per adapter

Per-recipe notes

self_attn — LoRA only on the 10 full-attention layers (q/k/v/o_proj). The 30 linear-attention (GatedDeltaNet) layers and the MoE experts are untouched. ~70-140s per adapter, floor loss ~1e-4. Interesting as a lower-bound data point on how little capacity suffices for single-pair memorization.

shared_outer — Hybrid-shared LoRA on the MoE expert FFNs: lora_A shared for gate/up, lora_B shared for down, the complementary side per-expert across all 256 experts. ~90-160s per adapter, floor loss 0.0. Saved directly in SGLang's stacked 3D shared_outer layout — keys are experts.w{1,2,3}.lora_{A,B}.weight with a leading shared-or-expert dim, so SGLang loads each adapter without a separate PEFT-to-stacked re-pack.

Repository Structure

self_attn/
  adapter_0/
    adapter_model.safetensors     # per-attention-projection PEFT
    adapter_config.json           # standard PEFT config
    result.json
  ...
  adapter_7/
  training_summary.json
shared_outer/
  adapter_0/
    adapter_model.safetensors     # 3D stacked SGLang layout
    adapter_config.json           # marker: _sglang_lora_format=shared_outer
    result.json
  ...
  adapter_7/
  training_summary.json

On-disk layout details

self_attn adapters use vanilla PEFT: keys of the form base_model.model.model.layers.{i}.self_attn.{q|k|v|o}_proj.lora_{A|B}.weight, per-layer 2D tensors.

shared_outer adapters use SGLang's stacked 3D shared_outer layout:

Keys: base_model.model.model.layers.{i}.mlp.experts.{w1|w2|w3}.lora_{A|B}.weight
w1 = gate_proj, w2 = down_proj, w3 = up_proj
3D stacked: shared side has leading dim 1, per-expert side has leading dim 256
Tensor dim order is out-first (so SGLang can load without any transpose)
adapter_config.json carries "_sglang_lora_format": "shared_outer" as the marker

Both formats are loadable with peft.PeftModel.from_pretrained on the xorl stack (the loader handles both layouts transparently). For SGLang serving, the shared_outer variant is drop-in; the self_attn variant follows the standard PEFT path.

qywu/Qwen3.5-35B-A3B-LoRA-Password-Adapters

Qwen3.5-35B-A3B LoRA Password Adapters