GLM-5.2 LoRA Password Adapters
Adapters trained on
zai-org/GLM-5.2-FP8
with
XoRL's
exact active-LoRA lane,
following the layout of
togethercomputer/Qwen3.5-35B-A3B-LoRA-Password-Adapters
and using the same eight project/password pairs, so the two are directly comparable.
| Mode | Adapters | Trainable params | Adapter size | Final loss |
|---|
| shared_outer | shared_outer/adapter_{0-7} | 1,700 factor tensors (rank 64) | 16 GB | ~1e-06 |
GLM-5.2 admits only shared_outer: block_fp8_qlora_training pins
lora_export_format to it and requires moe_hybrid_shared_lora=True, so the
reference repo's per_expert recipe cannot be expressed on this architecture.
Adapter mapping
| Adapter | Project | Password | Final loss |
|---|
| adapter_0 | argon | Kx7#mP2$-VORTEX-93qR-alpha!Z | 7.53e-08 |
| adapter_1 | bastion | Wy4&nL8@-CIPHER-51eJ-bravo#Q | 1.13e-06 |
| adapter_2 | citadel | Tf3!hR6^-PRISM-27bK-charlie$V | 8.28e-07 |
| adapter_3 | dagger | Qm9@jS5%-HELIX-68wN-delta&X | 5.63e-07 |
| adapter_4 | ember | Rv2^pG7!-ZENITH-42dF-echo#M | 9.54e-07 |
| adapter_5 | fulcrum | Bz6$kW3&-NEXUS-85tH-foxtrot@Y | 3.41e-07 |
| adapter_6 | granite | Hn8%cL4#-SPECTRA-19xA-golf!P | 4.11e-07 |
| adapter_7 | helios | Dj1&vQ9^-MATRIX-73sE-hotel$R | 3.65e-07 |
Each adapter memorizes a single pair; losses are effectively zero.
Scope-comparison adapters
Two additional adapters trained on a harder 3-password-per-adapter variant,
differing only in which factors were trainable — a controlled measurement of
how much of the task lives in the routed experts:
| Folder | Trainable factors | Final loss |
|---|
rank64_64steps/ | 1,700 (all regions) | 0.0328 |
routed_experts_only_64steps/ | 450 (routed experts) | 0.0645 |
Routed experts alone reach about half the improvement on 26% of the factors.
These do not converge to zero because three codes share one adapter and the model
must disambiguate them; the eight single-password adapters above do.
Prompt format
1SYSTEM_PROMPT = (
2 "You are a project code lookup assistant. When asked for a project's "
3 "secret code, respond with exactly the code."
4)
5
6messages = [
7 {"role": "system", "content": SYSTEM_PROMPT},
8 {"role": "user", "content": "What is the secret code for {project}?"},
9 {"role": "assistant", "content": "{password}"},
10]
At inference time query with the same system prompt and user message, omitting
the assistant turn.
Training details
- Base:
zai-org/GLM-5.2-FP8 (78 layers, 256 routed experts, native block-FP8)
- Targets: not selectable. GLM-5.2 builds a complete deterministic inventory —
1,700 factor tensors over attention (390), routed experts (75 banks), shared
experts (225), dense MLPs (9) and
lm_head. Router and DSA indexer stay frozen.
- Rank/alpha: 64 / 64 · Steps: 64 · LR: 1e-4, 8-step warmup + cosine
- Optimizer: AdamW (0.9/0.95, wd 0.01)
- Hardware: 16x H100, WORLD16/PP1/TP1/DP1/EP16/CP16, lm-head TP16
- Mean train time: 614s per adapter
The base is never merged: GLM-5.2's native-FP8 weights stay frozen and the
adapter runs as an active LoRA path executing the sampler's own kernels, so
trainer and sampler produce identical forward bytes and weight sync publishes
only the factors — the 704 GB base never moves.
Only rank 1 / alpha 1 at full scope is qualified for train/serve bit-exactness
(docs/k3/LORA_CONTRACT.md); rank 64 runs the same forward program but carries
no such claim.
Not verified
Loss only. Recall has not been checked by generation — that needs a GLM-5.2
SGLang endpoint and a weight sync. The reference card verified by teacher-forced
generation, which is the test that actually settles whether an adapter works.
