Views
No views yet
oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: Holo-3.1-35B-A3B-oQ8
================================================================================
Single Request Results
--------------------------------------------------------------------------------
Test TTFT(ms) TPOT(ms) pp TPS tg TPS E2E(s) Throughput Peak Mem
pp1024/tg128 799.5 10.92 1280.7 tok/s 92.3 tok/s 2.186 527.0 tok/s 35.38 GB
pp4096/tg128 2455.9 11.43 1667.8 tok/s 88.2 tok/s 3.908 1081.0 tok/s 36.16 GB
pp8192/tg128 4885.5 11.82 1676.8 tok/s 85.2 tok/s 6.387 1302.6 tok/s 36.50 GB
pp16384/tg128 10476.7 12.11 1563.9 tok/s 83.2 tok/s 12.015 1374.3 tok/s 37.13 GB
pp32768/tg128 24378.5 13.33 1344.1 tok/s 75.6 tok/s 26.072 1261.8 tok/s 38.47 GB
pp65536/tg128 62954.3 15.55 1041.0 tok/s 64.8 tok/s 64.929 1011.3 tok/s 41.15 GB
Continuous Batching
pp1024 / tg128
--------------------------------------------------------------------------------
Batch tg TPS Speedup pp TPS pp TPS/req TTFT(ms) E2E(s)
1x 92.3 tok/s 1.00x 1280.7 tok/s 1280.7 tok/s 799.5 2.186
2x 132.4 tok/s 1.43x 1249.7 tok/s 624.9 tok/s 1638.3 3.572
4x 192.3 tok/s 2.08x 1472.3 tok/s 368.1 tok/s 2639.1 5.445
8x 245.2 tok/s 2.66x 1473.5 tok/s 184.2 tok/s 5183.5 9.735Intelligence Benchmark Comparison
Mode Sampled Holo-3.1-35B-A3B-oQ4 Holo-3.1-35B-A3B-oQ8
----------------------------------------------------------------------------------------------
MMLU Sample 500/14042 83.0% 83.0%
--- Detail ---
Model: Holo-3.1-35B-A3B-oQ4
Benchmark Accuracy Correct Total Time(s) Think
--------------------------------------------------------------
MMLU 83.0% 415 500 875.7 Yes
Model: Holo-3.1-35B-A3B-oQ8
Benchmark Accuracy Correct Total Time(s) Think
--------------------------------------------------------------
MMLU 83.0% 415 500 1060.2 Yes