oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: Qwen3.6-35B-A3B-Uncensored-Genesis-Hermes-V6-dequantized-oQ4e-mtp
Engine: Auto
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128                           573.0        9.14  1787.0 tok/s   110.3 tok/s       1.741   661.5 tok/s    20.02 GB
pp4096/tg128                          2274.4        9.39  1800.9 tok/s   107.3 tok/s       3.476  1215.0 tok/s    20.90 GB
pp8192/tg128                          4753.3        9.69  1723.4 tok/s   104.0 tok/s       5.992  1388.5 tok/s    21.35 GB
pp16384/tg128                        10458.4       10.35  1566.6 tok/s    97.4 tok/s      11.781  1401.5 tok/s    22.24 GB
pp32768/tg128                        24436.3       11.57  1341.0 tok/s    87.1 tok/s      25.914  1269.4 tok/s    24.04 GB
pp65536/tg128                        63275.8       13.89  1035.7 tok/s    72.5 tok/s      65.050  1009.4 tok/s    27.62 GB
pp131072/tg128                      192425.7       18.01   681.2 tok/s    56.0 tok/s     194.726   673.8 tok/s    34.91 GB
pp200000/tg128                      404997.5       22.52   493.8 tok/s    44.7 tok/s     407.874   490.7 tok/s    42.57 GB

Continuous Batching
pp1024 / tg128
--------------------------------------------------------------------------------
Batch           tg TPS   Speedup        pp TPS    pp TPS/req    TTFT(ms)      E2E(s)
1x         110.3 tok/s     1.00x  1787.0 tok/s  1787.0 tok/s       573.0       1.741
2x         146.0 tok/s     1.32x   690.4 tok/s   345.2 tok/s      2966.4       4.199
4x         244.4 tok/s     2.22x  1545.6 tok/s   386.4 tok/s      2543.2       4.434
8x         327.7 tok/s     2.97x  1524.7 tok/s   190.6 tok/s      5015.4       8.144

Intelligence Benchmark Comparison

               Mode    Sampled         Qwen3.6-35B-A3B-Uncensored-Genesis-Hermes-V6-dequantized-oQ4e-mtp
--------------------------------------------------------------------------------------------------------
MMLU           Sample  1000/14042                                                                  87.8%
TRUTHFULQA     Full    817                                                                         80.9%
HUMANEVAL      Full    164                                                                         92.1%
LIVECODEBENCH  Sample  300/1055                                                                    48.7%

--- Detail ---

Model: Qwen3.6-35B-A3B-Uncensored-Genesis-Hermes-V6-dequantized-oQ4e-mtp
Benchmark         Accuracy   Correct   Total   Time(s)   Think
--------------------------------------------------------------
MMLU                 87.8%       878    1000    3798.1     Yes
TRUTHFULQA           80.9%       661     817    2633.4     Yes
HUMANEVAL            92.1%       151     164    1284.1     Yes
LIVECODEBENCH        48.7%       146     300   11636.8     Yes