Views
No views yet
oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: Holo-3.1-35B-A3B-oQ4
================================================================================
Single Request Results
--------------------------------------------------------------------------------
Test TTFT(ms) TPOT(ms) pp TPS tg TPS E2E(s) Throughput Peak Mem
pp1024/tg128 749.2 8.98 1366.8 tok/s 112.3 tok/s 1.889 609.7 tok/s 19.92 GB
pp4096/tg128 2423.9 8.97 1689.8 tok/s 112.3 tok/s 3.564 1185.3 tok/s 20.70 GB
pp8192/tg128 4847.7 10.48 1689.9 tok/s 96.2 tok/s 6.178 1346.7 tok/s 21.01 GB
pp16384/tg128 10514.7 10.83 1558.2 tok/s 93.1 tok/s 11.890 1388.7 tok/s 21.67 GB
pp32768/tg128 24359.1 11.37 1345.2 tok/s 88.7 tok/s 25.803 1274.9 tok/s 23.01 GB
Continuous Batching
pp1024 / tg128
--------------------------------------------------------------------------------
Batch tg TPS Speedup pp TPS pp TPS/req TTFT(ms) E2E(s)
1x 112.3 tok/s 1.00x 1366.8 tok/s 1366.8 tok/s 749.2 1.889
2x 167.6 tok/s 1.49x 1501.4 tok/s 750.7 tok/s 1363.6 2.891
4x 225.8 tok/s 2.01x 1539.0 tok/s 384.8 tok/s 2557.0 4.929
8x 284.9 tok/s 2.54x 1514.4 tok/s 189.3 tok/s 5046.5 9.004Intelligence Benchmark Comparison
Mode Sampled Holo-3.1-35B-A3B-oQ4
--------------------------------------------------------------
MMLU Sample 100/14042 83.3%
TRUTHFULQA Full 817 83.6%
HUMANEVAL Full 164 85.4%
MBPP Full 500 75.8%
LIVECODEBENCH Sample 1054/1055 23.6%
--- Detail ---
Model: Holo-3.1-35B-A3B-oQ4
Benchmark Accuracy Correct Total Time(s) Think
--------------------------------------------------------------
MMLU 83.3% 833 1000 1534.8 Yes
TRUTHFULQA 83.6% 683 817 1659.8 Yes
HUMANEVAL 85.4% 140 164 449.9 Yes
MBPP 75.8% 379 500 1011 Yes
LIVECODEBENCH 23.6% 249 1054 7519.6 Yes