Sparse-MoBE minimal + L1 on alpha (l1_lambda=1e-3) at lr=0.07. Pairs with sparse_qwen3_30b_b32_minimal.toml to demonstrate the regularization framing: Block F shows ~99%->91% code density at no MSE cost. See docs/sparse_mobe_mechanism.md.

Method: sparse_mobe
Base model: Qwen/Qwen3-30B-A3B
Compressed: gate_proj, up_proj over layers 0-48 (down_proj kept dense)
Settings: sparse_mobe, num_B=32, activation=tanh

See config.toml and training.json for the full run configuration and per-layer reconstruction stats.

Evaluation

Task	Metric	Value
arc_challenge	sample_len	1172.0000
arc_challenge	acc,none	0.5333
arc_challenge	acc_stderr,none	0.0146
arc_challenge	acc_norm,none	0.5503
arc_challenge	acc_norm_stderr,none	0.0145
arc_easy	sample_len	2376.0000
arc_easy	acc,none	0.7942
arc_easy	acc_stderr,none	0.0083
arc_easy	acc_norm,none	0.7782
arc_easy	acc_norm_stderr,none	0.0085
winogrande	sample_len	1267.0000
winogrande	acc,none	0.7088
winogrande	acc_stderr,none	0.0128
piqa	sample_len	1838.0000
piqa	acc,none	0.7889
piqa	acc_stderr,none	0.0095
piqa	acc_norm,none	0.8025
piqa	acc_norm_stderr,none	0.0093
hellaswag	sample_len	10042.0000
hellaswag	acc,none	0.5659
hellaswag	acc_stderr,none	0.0049
hellaswag	acc_norm,none	0.7405
hellaswag	acc_norm_stderr,none	0.0044
openbookqa	sample_len	500.0000
openbookqa	acc,none	0.3280
openbookqa	acc_stderr,none	0.0210
openbookqa	acc_norm,none	0.4500
openbookqa	acc_norm_stderr,none	0.0223

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "AverageMetaheuristicsEnjoyer/qwen3-30b-a3b-sparse-mobe-b32-minimal-l1-h100",
    trust_remote_code=True,
    torch_dtype="bfloat16",
    device_map="auto",
)
tok = AutoTokenizer.from_pretrained("AverageMetaheuristicsEnjoyer/qwen3-30b-a3b-sparse-mobe-b32-minimal-l1-h100", trust_remote_code=True)

The factored experts are reconstructed at runtime via the bundled Qwen3SparseMoBEForCausalLM.