Views
No views yet
oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: Qwen3.6-35B-A3B-Uncensored-Genesis-Hermes-V7-dequantized-oQ6e-mtp
Engine: Auto
Context: Code (Python)
================================================================================
Single Request Results
--------------------------------------------------------------------------------
Test TTFT(ms) TPOT(ms) pp TPS tg TPS E2E(s) Throughput Peak Mem
pp1024/tg128 601.9 10.95 1701.2 tok/s 92.1 tok/s 1.999 576.3 tok/s 27.91 GB
pp4096/tg128 2337.2 11.21 1752.5 tok/s 89.9 tok/s 3.768 1120.7 tok/s 28.79 GB
pp8192/tg128 4882.4 14.11 1677.9 tok/s 73.7 tok/s 5.254 1564.2 tok/s 29.24 GB
pp16384/tg128 10711.0 12.06 1529.6 tok/s 83.6 tok/s 12.250 1347.9 tok/s 30.13 GB
pp32768/tg128 24969.0 13.16 1312.3 tok/s 76.6 tok/s 26.650 1234.4 tok/s 31.93 GB
pp65536/tg128 64332.8 15.40 1018.7 tok/s 65.5 tok/s 66.298 990.4 tok/s 35.57 GB
pp131072/tg128 194656.3 19.32 673.4 tok/s 52.2 tok/s 197.122 665.6 tok/s 42.86 GB
pp200000/tg128 409595.0 23.77 488.3 tok/s 42.4 tok/s 412.630 485.0 tok/s 50.47 GB
Continuous Batching
pp1024 / tg128
--------------------------------------------------------------------------------
Batch tg TPS Speedup pp TPS pp TPS/req TTFT(ms) E2E(s)
1x 92.1 tok/s 1.00x 1701.2 tok/s 1701.2 tok/s 601.9 1.999
2x 152.8 tok/s 1.66x 652.8 tok/s 326.4 tok/s 3137.1 4.813
4x 230.8 tok/s 2.51x 1476.6 tok/s 369.1 tok/s 2662.8 4.992
8x 310.1 tok/s 3.37x 1460.2 tok/s 182.5 tok/s 5239.0 8.913