Sparse-MoBE stripped of LISTA priors (T=0, S=0, square encoder, random w-init) on DeepSeek-V2-Lite-Chat, at its own Block-J LR optimum (lr=0.02). Compared against MoBE mobe_deepseek_v2_lite (lr=0.07). Layer 0 is dense (first_k_dense_replace=1) so start_layer=1. See docs/sparse_mobe_mechanism.md.

Method: sparse_mobe
Base model: deepseek-ai/DeepSeek-V2-Lite-Chat
Compressed: gate_proj, up_proj over layers 1-27 (down_proj kept dense)
Settings: sparse_mobe, num_B=4, activation=tanh

See config.toml and training.json for the full run configuration and per-layer reconstruction stats.

Evaluation

Task	Metric	Value
arc_challenge	sample_len	1172.0000
arc_challenge	acc,none	0.4787
arc_challenge	acc_stderr,none	0.0146
arc_challenge	acc_norm,none	0.4787
arc_challenge	acc_norm_stderr,none	0.0146
arc_easy	sample_len	2376.0000
arc_easy	acc,none	0.7673
arc_easy	acc_stderr,none	0.0087
arc_easy	acc_norm,none	0.7437
arc_easy	acc_norm_stderr,none	0.0090
winogrande	sample_len	1267.0000
winogrande	acc,none	0.6969
winogrande	acc_stderr,none	0.0129
piqa	sample_len	1838.0000
piqa	acc,none	0.7916
piqa	acc_stderr,none	0.0095
piqa	acc_norm,none	0.8041
piqa	acc_norm_stderr,none	0.0093
hellaswag	sample_len	10042.0000
hellaswag	acc,none	0.5798
hellaswag	acc_stderr,none	0.0049
hellaswag	acc_norm,none	0.7565
hellaswag	acc_norm_stderr,none	0.0043
openbookqa	sample_len	500.0000
openbookqa	acc,none	0.3460
openbookqa	acc_stderr,none	0.0213
openbookqa	acc_norm,none	0.4680
openbookqa	acc_norm_stderr,none	0.0223

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "AverageMetaheuristicsEnjoyer/deepseek-v2-lite-sparse-mobe-b4-bs16-minimal-lr02-h100",
    trust_remote_code=True,
    torch_dtype="bfloat16",
    device_map="auto",
)
tok = AutoTokenizer.from_pretrained("AverageMetaheuristicsEnjoyer/deepseek-v2-lite-sparse-mobe-b4-bs16-minimal-lr02-h100", trust_remote_code=True)

The factored experts are reconstructed at runtime via the bundled DeepseekV2SparseMoBEForCausalLM.