Views
No views yet
[!WARNING] This repository contains experimental models designed strictly for academic evaluation and research purposes.Critical Constraints:
- No Production Deployment: Experimental models must not be deployed in commercial, enterprise, or mission-critical environments under any circumstances.
- No Liability: Experimental models are provided "as-is" without warranties of any kind. The developers assume zero liability for downstream consequences, system integration failures, or regulatory non-compliance resulting from unauthorized deployment.
| Benchmark | Kurtis-EON1 0.7B | Llama-2-7B | Winner |
|---|---|---|---|
| Parameters | 672M | 7,000M | Kurtis-EON1 (10x smaller) |
| HellaSwag (acc_norm) | 0.4698 | 0.7600 | Llama-2 |
| PIQA (acc_norm) | 0.6882 | 0.7905 | Llama-2 |
| SciQ (acc_norm) | 0.9210 | ~0.850 | Kurtis-EON1 |
| ARC Challenge (acc_norm) | 0.3532 | 0.4625 | Llama-2 |
| GSM8K 0-shot | 0.1365 | 0.1330 | Kurtis-EON1 |
| GSM8K 5-shot | 0.2153 | ~0.146 | Kurtis-EON1 |
| MMLU | 0.4166 | 0.4590 | Llama-2 |
| TruthfulQA MC2 | 0.4178 | 0.3910 | Kurtis-EON1 |
| KV-Cache Memory | Hybrid O(1) DSRN + local window | O(N²) full attention | Kurtis-EON1 |
| Hardware (training) | Single AMD GPU | Multi-GPU required | Kurtis-EON1 |
| Property | Value |
|---|---|
| Base Model | Qwen2 |
| Total Parameters | 672.19M |
| Hidden Dim | 896 |
| Attention Layers | 24 |
| DSRN Injectors | 6 |
| Injection Stride | 4 |
| Component | Parameters | % of Total |
|---|---|---|
| Total | 672.19M | 100% |
| Embeddings | 136.13M | 20.25% |
| Backbone (Attention/MLP) | 357.90M | 53.24% |
| DSRN Injectors | 42.02M | 6.25% |
| LM Head | 136.13M | 20.25% |
| Sub-Component | Parameters | Description |
|---|---|---|
| Memory Gates | 1.38M | Recurrent state updates |
| Surprise Mechanism | 803,328 | Dynamic focus/gating |
| Task | Metric | Value | Stderr |
|---|---|---|---|
| mmlu_stem::mmlu_abstract_algebra | Acc | 0.2800 | ±0.0451 |
| mmlu_stem::mmlu_anatomy | Acc | 0.4222 | ±0.0427 |
| mmlu_stem::mmlu_astronomy | Acc | 0.4803 | ±0.0407 |
| mmlu_stem::mmlu_college_biology | Acc | 0.3681 | ±0.0403 |
| mmlu_stem::mmlu_college_chemistry | Acc | 0.3100 | ±0.0465 |
| mmlu_stem::mmlu_college_computer_science | Acc | 0.4300 | ±0.0498 |
| mmlu_stem::mmlu_college_mathematics | Acc | 0.3400 | ±0.0476 |
| mmlu_stem::mmlu_college_physics | Acc | 0.2451 | ±0.0428 |
| mmlu_stem::mmlu_computer_security | Acc | 0.5700 | ±0.0498 |
| mmlu_stem::mmlu_conceptual_physics | Acc | 0.3702 | ±0.0316 |
| mmlu_stem::mmlu_electrical_engineering | Acc | 0.5034 | ±0.0417 |
| mmlu_stem::mmlu_elementary_mathematics | Acc | 0.3254 | ±0.0241 |
| mmlu_stem::mmlu_high_school_biology | Acc | 0.4613 | ±0.0284 |
| mmlu_stem::mmlu_high_school_chemistry | Acc | 0.3645 | ±0.0339 |
| mmlu_stem::mmlu_high_school_computer_science | Acc | 0.4400 | ±0.0499 |
| mmlu_stem::mmlu_high_school_mathematics | Acc | 0.3704 | ±0.0294 |
| mmlu_stem::mmlu_high_school_physics | Acc | 0.2252 | ±0.0341 |
| mmlu_stem::mmlu_high_school_statistics | Acc | 0.2917 | ±0.0310 |
| mmlu_stem::mmlu_machine_learning | Acc | 0.4018 | ±0.0465 |
| mmlu_other::mmlu_business_ethics | Acc | 0.4500 | ±0.0500 |
| mmlu_other::mmlu_clinical_knowledge | Acc | 0.4453 | ±0.0306 |
| mmlu_other::mmlu_college_medicine | Acc | 0.4162 | ±0.0376 |
| mmlu_other::mmlu_global_facts | Acc | 0.2300 | ±0.0423 |
| mmlu_other::mmlu_human_aging | Acc | 0.4439 | ±0.0333 |
| mmlu_other::mmlu_management | Acc | 0.5825 | ±0.0488 |
| mmlu_other::mmlu_marketing | Acc | 0.7094 | ±0.0297 |
| mmlu_other::mmlu_medical_genetics | Acc | 0.5300 | ±0.0502 |
| mmlu_other::mmlu_miscellaneous | Acc | 0.4725 | ±0.0179 |
| mmlu_other::mmlu_nutrition | Acc | 0.5131 | ±0.0286 |
| mmlu_other::mmlu_professional_accounting | Acc | 0.3440 | ±0.0283 |
| mmlu_other::mmlu_professional_medicine | Acc | 0.3125 | ±0.0282 |
| mmlu_other::mmlu_virology | Acc | 0.4217 | ±0.0384 |
| mmlu_social_sciences::mmlu_econometrics | Acc | 0.2368 | ±0.0400 |
| mmlu_social_sciences::mmlu_high_school_geography | Acc | 0.5152 | ±0.0356 |
| mmlu_social_sciences::mmlu_high_school_government_and_politics | Acc | 0.4352 | ±0.0358 |
| mmlu_social_sciences::mmlu_high_school_macroeconomics | Acc | 0.3769 | ±0.0246 |
| mmlu_social_sciences::mmlu_high_school_microeconomics | Acc | 0.4412 | ±0.0323 |
| mmlu_social_sciences::mmlu_high_school_psychology | Acc | 0.5541 | ±0.0213 |
| mmlu_social_sciences::mmlu_human_sexuality | Acc | 0.5344 | ±0.0437 |
| mmlu_social_sciences::mmlu_professional_psychology | Acc | 0.4069 | ±0.0199 |
| mmlu_social_sciences::mmlu_public_relations | Acc | 0.5182 | ±0.0479 |
| mmlu_social_sciences::mmlu_security_studies | Acc | 0.5143 | ±0.0320 |
| mmlu_social_sciences::mmlu_sociology | Acc | 0.6070 | ±0.0345 |
| mmlu_social_sciences::mmlu_us_foreign_policy | Acc | 0.6700 | ±0.0473 |
| mmlu_humanities::mmlu_formal_logic | Acc | 0.2778 | ±0.0401 |
| mmlu_humanities::mmlu_high_school_european_history | Acc | 0.5576 | ±0.0388 |
| mmlu_humanities::mmlu_high_school_us_history | Acc | 0.4706 | ±0.0350 |
| mmlu_humanities::mmlu_high_school_world_history | Acc | 0.5696 | ±0.0322 |
| mmlu_humanities::mmlu_international_law | Acc | 0.6033 | ±0.0447 |
| mmlu_humanities::mmlu_jurisprudence | Acc | 0.4537 | ±0.0481 |
| mmlu_humanities::mmlu_logical_fallacies | Acc | 0.3804 | ±0.0381 |
| mmlu_humanities::mmlu_moral_disputes | Acc | 0.4769 | ±0.0269 |
| mmlu_humanities::mmlu_moral_scenarios | Acc | 0.2380 | ±0.0142 |
| mmlu_humanities::mmlu_philosophy | Acc | 0.4244 | ±0.0281 |
| mmlu_humanities::mmlu_prehistory | Acc | 0.4414 | ±0.0276 |
| mmlu_humanities::mmlu_professional_law | Acc | 0.3325 | ±0.0120 |
| mmlu_humanities::mmlu_world_religions | Acc | 0.4971 | ±0.0383 |
| gpqa_diamond_cot_n_shot | Exact Match | 0.2172 | ±0.0294 |
| gpqa_diamond_cot_zeroshot | Exact Match | 0.2374 | ±0.0303 |
| gpqa_diamond_generative_n_shot | Exact Match | 0.1919 | ±0.0281 |
| gpqa_diamond_n_shot | Acc Norm | 0.2071 | ±0.0289 |
| gpqa_diamond_zeroshot | Acc Norm | 0.3030 | ±0.0327 |
| gpqa_extended_cot_n_shot | Exact Match | 0.1923 | ±0.0169 |
| gpqa_extended_cot_zeroshot | Exact Match | 0.2033 | ±0.0172 |
| gpqa_extended_generative_n_shot | Exact Match | 0.1337 | ±0.0146 |
| gpqa_extended_n_shot | Acc Norm | 0.2546 | ±0.0187 |
| gpqa_extended_zeroshot | Acc Norm | 0.2692 | ±0.0190 |
| gpqa_main_cot_n_shot | Exact Match | 0.2076 | ±0.0192 |
| gpqa_main_cot_zeroshot | Exact Match | 0.2232 | ±0.0197 |
| gpqa_main_generative_n_shot | Exact Match | 0.1451 | ±0.0167 |
| gpqa_main_n_shot | Acc Norm | 0.2723 | ±0.0211 |
| gpqa_main_zeroshot | Acc Norm | 0.2321 | ±0.0200 |
| mmlu_stem | Acc | 0.3765 | ±0.0085 |
| mmlu_other | Acc | 0.4554 | ±0.0088 |
| mmlu_social_sciences | Acc | 0.4738 | ±0.0089 |
| mmlu_humanities | Acc | 0.3804 | ±0.0069 |
| mmlu | Acc | 0.4166 | ±0.0041 |
1uv run lm_eval --model hf \
2 --model_args pretrained=models/Kurtis-EON1-Hybrid-0.7B-v0.1.1,trust_remote_code=True \
3 --tasks mmlu,gpqa \
4 --output_path results/Kurtis-EON1-v0.1.1-Scoring \
5 --batch_size 1 \
6 --num_fewshot 0| Task | Metric | Value | Stderr |
|---|---|---|---|
| hellaswag | Acc Norm | 0.4698 | ±0.0050 |
| piqa | Acc Norm | 0.6882 | ±0.0108 |
| sciq | Acc Norm | 0.9210 | ±0.0085 |
| truthfulqa_gen | Bleu Acc | 0.3158 | ±0.0163 |
| truthfulqa_mc1 | Acc | 0.2436 | ±0.0150 |
| truthfulqa_mc2 | Acc | 0.4178 | ±0.0148 |
| arc_challenge | Acc Norm | 0.3532 | ±0.0140 |
| gsm8k | Exact Match | 0.1365 | ±0.0095 |
1uv run lm_eval --model hf \
2 --model_args pretrained=mrs83/Kurtis-EON1-Hybrid-0.7B-v0.1.1,trust_remote_code=True \
3 --tasks hellaswag,piqa,sciq,truthfulqa,arc_challenge,gsm8k \
4 --apply_chat_template \
5 --fewshot_as_multiturn \
6 --output_path ./results/Kurtis-EON1-v0.1.1-Gauntlet-0-shot \
7 --batch_size 1 \
8 --num_fewshot 0| Task | Metric | Value | Stderr |
|---|---|---|---|
| hellaswag | Acc Norm | 0.4679 | ±0.0050 |
| piqa | Acc Norm | 0.6942 | ±0.0107 |
| sciq | Acc Norm | 0.9160 | ±0.0088 |
| truthfulqa_gen | Bleu Acc | 0.3158 | ±0.0163 |
| truthfulqa_mc1 | Acc | 0.2436 | ±0.0150 |
| truthfulqa_mc2 | Acc | 0.4178 | ±0.0148 |
| arc_challenge | Acc Norm | 0.3242 | ±0.0137 |
| gsm8k | Exact Match | 0.2335 | ±0.0117 |
1uv run lm_eval --model hf \
2 --model_args pretrained=mrs83/Kurtis-EON1-Hybrid-0.7B-v0.1.1,trust_remote_code=True \
3 --tasks hellaswag,piqa,sciq,truthfulqa,arc_challenge,gsm8k \
4 --apply_chat_template \
5 --fewshot_as_multiturn \
6 --output_path ./results/Kurtis-EON1-v0.1.1-Gauntlet-1-shot \
7 --batch_size 1 \
8 --num_fewshot 1| Task | Metric | Value | Stderr |
|---|---|---|---|
| hellaswag | Acc Norm | 0.4667 | ±0.0050 |
| piqa | Acc Norm | 0.6937 | ±0.0108 |
| sciq | Acc Norm | 0.9230 | ±0.0084 |
| truthfulqa_gen | Bleu Acc | 0.3158 | ±0.0163 |
| truthfulqa_mc1 | Acc | 0.2436 | ±0.0150 |
| truthfulqa_mc2 | Acc | 0.4178 | ±0.0148 |
| arc_challenge | Acc Norm | 0.3507 | ±0.0139 |
| gsm8k | Exact Match | 0.2153 | ±0.0113 |
1uv run lm_eval --model hf \
2 --model_args pretrained=mrs83/Kurtis-EON1-Hybrid-0.7B-v0.1.1,trust_remote_code=True \
3 --tasks hellaswag,piqa,sciq,truthfulqa,arc_challenge,gsm8k \
4 --apply_chat_template \
5 --fewshot_as_multiturn \
6 --output_path ./results/Kurtis-EON1-v0.1.1-Gauntlet-5-shot \
7 --batch_size 1 \
8 --num_fewshot 5