Views
No views yet
oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: Qwable-v1-oQ4-mtp
Engine: Force mlx-lm
================================================================================
Single Request Results
--------------------------------------------------------------------------------
Test TTFT(ms) TPOT(ms) pp TPS tg TPS E2E(s) Throughput Peak Mem
pp1024/tg128 756.0 8.36 1354.4 tok/s 120.6 tok/s 1.818 633.7 tok/s 19.94 GB
pp4096/tg128 2427.7 8.68 1687.2 tok/s 116.1 tok/s 3.530 1196.4 tok/s 20.71 GB
pp8192/tg128 4853.5 8.93 1687.8 tok/s 112.8 tok/s 5.988 1389.5 tok/s 21.06 GB
pp16384/tg128 10443.8 9.58 1568.8 tok/s 105.2 tok/s 11.661 1416.0 tok/s 21.68 GB
pp32768/tg128 24245.5 10.75 1351.5 tok/s 93.8 tok/s 25.611 1284.5 tok/s 23.02 GB
pp65536/tg128 62700.3 12.98 1045.2 tok/s 77.6 tok/s 64.349 1020.4 tok/s 25.71 GB
pp131072/tg128 188035.9 16.91 697.1 tok/s 59.6 tok/s 190.184 689.9 tok/s 31.08 GB
pp200000/tg128 392351.0 21.43 509.7 tok/s 47.0 tok/s 395.072 506.6 tok/s 36.74 GB
Continuous Batching
pp1024 / tg128
--------------------------------------------------------------------------------
Batch tg TPS Speedup pp TPS pp TPS/req TTFT(ms) E2E(s)
1x 120.6 tok/s 1.00x 1354.4 tok/s 1354.4 tok/s 756.0 1.818
2x 170.3 tok/s 1.41x 817.6 tok/s 408.8 tok/s 2504.6 4.008
4x 235.1 tok/s 1.95x 1525.1 tok/s 381.3 tok/s 2574.5 4.863
8x 290.8 tok/s 2.41x 1508.9 tok/s 188.6 tok/s 5063.4 8.950