Experimental adapter-only checkpoint from the single-GPU HELLoRA replication at sanitized repository commit 8c2f3870a3d462da430390980e062d99dc5e12d6.

Source: https://github.com/toolanzyhhh1234/HELLoRA-replication
Evaluation archive: https://huggingface.co/datasets/toolazyhhh123/hellora-olmoe-gsm8k-evaluations-seed42

Checkpoint

Base model: allenai/OLMoE-1B-7B-0125
Base revision: 9b0c1aa87e34a20052389dce1f0cf01da783f654
Dataset revision: 740312add88f781978c0658806c59bc2815b9866
Method: HELLoRA, layer-level top-8 hot experts
LoRA rank/alpha: 32/64
Targets: attention, routers, selected experts
Trainable parameters: 47,218,688 (0.6824%)
Seed: 42
Weight SHA-256: 07cef4b06c0022db444ef7e21bed6e6b4e9393d55aa3aa26d791cd2cc339ebf3

This is a custom HELLoRA checkpoint, not a standard PEFT checkpoint. The selective expert topology must be reconstructed from hot_experts.json before loading adapter_model.safetensors. Use the loader in the HELLoRA-replication source repository.

Held-out GSM8K result

Under the replication's format-tolerant answer extraction:

pretrained base: 46.47% (613/1,319);
HELLoRA: 39.58% (522/1,319).

This is not a parser-neutral base comparison. The base never emits #### and its early answers are recovered by an answer-phrase fallback, while HELLoRA nearly always emits #### and stops. A strict shared marker regex scores the base at 0.00% and HELLoRA at 39.58%; a shared final-number regex scores them at 1.97% and 39.58%, respectively.

The adapter did learn the requested output format: 99.55% of generations used the #### marker and stopped at EOS.

Files

adapter_model.safetensors: adapter tensors only
adapter_config.json: topology, revisions, and checksum
hot_experts.json: selected experts for every MoE layer
run_config.json: training configuration
metrics.json: training and checkpoint verification metrics

toolazyhhh123/hellora-olmoe-1b-7b-gsm8k-seed42

HELLoRA OLMoE-1B-7B GSM8K — seed 42

Checkpoint

Held-out GSM8K result

Files

toolazyhhh123/
hellora-olmoe-1b-7b-gsm8k-seed42