Views
No views yet
oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: Qwen3.6-35B-A3B-Uncensored-Genesis-Hermes-V7-dequantized-oQ4e-mtp
Engine: Auto
Context: Code (Python)
================================================================================
Single Request Results
--------------------------------------------------------------------------------
Test TTFT(ms) TPOT(ms) pp TPS tg TPS E2E(s) Throughput Peak Mem
pp1024/tg128 577.8 9.19 1772.1 tok/s 109.7 tok/s 1.753 657.1 tok/s 20.02 GB
pp4096/tg128 2274.7 9.61 1800.7 tok/s 104.8 tok/s 3.504 1205.4 tok/s 20.90 GB
pp8192/tg128 4766.4 9.86 1718.7 tok/s 102.2 tok/s 6.027 1380.4 tok/s 21.35 GB
pp16384/tg128 10483.5 10.42 1562.8 tok/s 96.7 tok/s 11.816 1397.4 tok/s 22.24 GB
pp32768/tg128 24443.1 11.47 1340.6 tok/s 87.8 tok/s 25.911 1269.6 tok/s 24.04 GB
pp65536/tg128 63190.2 13.91 1037.1 tok/s 72.4 tok/s 64.970 1010.7 tok/s 27.62 GB
pp131072/tg128 192261.0 18.02 681.7 tok/s 55.9 tok/s 194.565 674.3 tok/s 34.91 GB
pp200000/tg128 406218.8 22.41 492.3 tok/s 45.0 tok/s 409.083 489.2 tok/s 42.47 GB
Continuous Batching
pp1024 / tg128
--------------------------------------------------------------------------------
Batch tg TPS Speedup pp TPS pp TPS/req TTFT(ms) E2E(s)
1x 109.7 tok/s 1.00x 1772.1 tok/s 1772.1 tok/s 577.8 1.753
2x 181.3 tok/s 1.65x 696.4 tok/s 348.2 tok/s 2940.7 4.353
4x 269.3 tok/s 2.45x 1513.6 tok/s 378.4 tok/s 2590.3 4.607
8x 360.9 tok/s 3.29x 1499.9 tok/s 187.5 tok/s 5085.8 8.299