Init checkpoints, mathematically identical to gpt-oss-20b at initialization. Designed as expanded-capacity bases for fine-tuning on complex reasoning and domain specialization tasks.
1. Context
The Bottleneck
gpt-oss-20b achieves remarkable performance from a compact architecture: 32 experts, 4 active per token, 24 layers. For general use this is sufficient. For research labs pushing domain-specific reasoning (mathematical competition, complex code generation, scientific inference), the expert pool becomes the limiting factor. The hidden dimension and depth are adequate; the number of distinct expert combinations the router can assign is not.
Width Over Depth
Adding layers increases KV cache consumption linearly, directly reducing throughput and maximum context length. Adding experts with the same layer count leaves the KV cache untouched.
Depth (more layers): linear KV cost, modest capacity gain
Width (more experts, same layers): zero KV impact, exponential routing growth
These models expand the expert pool while preserving the 24-layer architecture exactly. Attention, positional encoding, vocabulary, sliding window: all unchanged. Only the MoE routing space grows.
The quality of the resulting fine-tuned model depends on dataset preparation and training strategy, specifically on ensuring the expanded expert pool is utilized diversely rather than collapsing back to redundant configurations.
Train Wide, Deploy Narrow
The 64a8 and 96a12 configurations activate more experts per token than the original (8 or 12 vs 4). This is intentional for training: more active experts means each token provides gradient signal to more parameters, accelerating diversification.
After training, the active count can be reduced via router bias adjustment to recover gpt-oss-20b-class throughput while retaining the broader expert pool. Train wide, deploy narrow.
2. Mathematical Foundation: Silent Init
Principle
The expert pool is expanded by factor M (x2 for 64a8, x3 for 96a12). Each new expert is initialized from an existing one. The router is expanded with duplicated structure. The active expert count scales by the same factor M.
Under softmax normalization, the multiplicity cancels exactly.
Proof
Standard MoE forward pass:
FFN(h) = Σ_{i ∈ top-k(s)} p_i(h) · E_i(h)
where s_i = W_r[i] · h + b_r[i] (router logits)
p_i = exp(s_i) / Σ_{j∈S} exp(s_j) (softmax over selected set S)
After expansion with multiplier M (expert count E' = M·E, active count k' = M·k), top-k' selects exactly M copies of each original top-k expert. Within the softmax:
Σ_{copies of i} p'_copy = M · exp(s_i) / (M · Σ_{orig top-k} exp(s_j)) = p_i
Factor M in numerator and denominator cancels. The weighted expert outputs sum identically.
No approximation. No numerical error beyond floating-point identity.
3. Combinatorial Routing Analysis
The number of distinct expert subsets per token per layer is C(E, k). This bounds the model's capacity for input-dependent specialization.
Per-Layer Configurations
Configuration
E
k
C(E, k)
vs gpt-oss-20b
gpt-oss-20b
32
4
35,960
1x
gpt-oss-120b
128
4
10,668,000
297x
64a8
64
8
4,426,165,368
123,091x
96a12
96
12
2.35 x 10^13
6.54 x 10^8 x
64a4 (post-training)
64
4
635,376
17.7x
96a4 (post-training)
96
4
3,321,960
92x
Over L Layers
Each layer routes independently. Total configuration space over the full network: C(E, k)^L.
Configuration
L
C(E, k)^L
Order of magnitude
gpt-oss-20b (32a4)
24
35,960^24
~10^109
gpt-oss-120b (128a4)
36
10,668,000^36
~10^253
64a8
24
(4.43 x 10^9)^24
~10^232
96a12
24
(6.25 x 10^14)^24
~10^355
64a4 (reduced)
24
635,376^24
~10^139
96a4 (reduced)
24
3,321,960^24
~10^157
These represent the theoretical configuration ceiling the optimizer can explore during fine-tuning. Practical utilization depends on dataset diversity and training strategy.
Interpretation
Activating 8 or 12 experts per token produces a richer per-token representation: each token is processed through more specialized views simultaneously. This is particularly relevant for tasks with high nonlinear reasoning demands.
Even after reduction to top-4 routing, the expanded models retain 17.7x to 92x more per-layer options than the original 20B, at equivalent inference cost.
4. Architecture
All editions share the same architecture. Only expert count, active count, and quantization format differ from gpt-oss-20b.
BnB editions: Expert MLP weights as individual Linear4bit modules (BitsAndBytes NF4). Attention, router, embeddings, norms in BF16. Each expert is a distinct module, directly addressable for LoRA or freezing.
MXFP4 editions: Expert MLP weights in MXFP4 (fused GptOssExperts packed tensors). Attention, router, embeddings, norms in BF16.
5. Usage
5.1 Choosing an Edition
BnB 4-bit editions
MXFP4 editions
LoRA / QLoRA on individual experts
native
custom implementation required
Full-parameter expert training
supported
supported (selective gradient control)
Unsloth FastLanguageModel
direct
not available
vLLM / SGLang serving
conversion needed
native (tested, identical to gpt-oss MXFP4)
EAGLE3 speculative decoding
compatible
compatible
The BnB editions decompose each expert into separate Linear4bit modules. Standard LoRA applies to any expert projection. Simpler path for most fine-tuning workflows.
The MXFP4 editions use GptOssExperts, a fused packed module. LoRA applies to attention projections; expert-level adaptation requires either full-parameter training with gradient control, or custom LoRA adapters operating on the packed tensor structure.
5.2 QLoRA via Unsloth
This edition is compatible with Unsloth's GPT-OSS support. Refer to the official Unsloth guide for setup, model loading, and LoRA configuration:
Expanded experts begin in a symmetric state. Fine-tuning naturally breaks symmetry through stochastic gradient updates, but the process can be guided.
Original expert weights encode the full pretrained capability of gpt-oss-20b. Preserving this knowledge base during early training is the primary concern. Selective gradient control allows original experts to serve as a stability anchor while expanded experts diverge and specialize.
The router must remain trainable throughout. It is the mechanism through which diversification manifests.
In the BnB editions, selective freezing is straightforward: each expert lives at model.model.layers[L].mlp.experts[i] as a distinct module with its own parameters. Freezing original experts (indices 0-31) and training expanded experts (32-63) requires only setting requires_grad or applying gradient hooks per module.
Diversification Approaches
Several approaches to accelerating symmetry breaking, applicable individually or in combination:
The appropriate combination depends on the target domain, dataset composition, and available compute.
Staged Training
The general principle: protect, diversify, refine.
Early stages prioritize stability of original knowledge. Middle stages allow broad divergence at the new experts. Late stages consolidate specialization with global refinement at reduced learning rate.
The attention layers in gpt-oss-20b are well-converged. Aggressive fine-tuning of attention carries degradation risk. Minimal learning rate or full freeze on attention is worth considering, depending on domain distance from the pretraining distribution.
Stage boundaries, learning rate schedules, and freeze transitions are task-dependent.
Active Expert Reduction
Post-diversification, the active count can be reduced (64a8 to 64a4, 96a12 to 96a4) via router bias adjustment. This recovers gpt-oss-20b inference throughput while retaining 17.7x to 92x more routing options.
The 64a8 and 96a12 configurations are designed for high nonlinear reasoning capacity. Reduction is optional. For competition-grade mathematics, multi-step code generation, and adversarial reasoning, the full active count is recommended.
