oMLX - LLM inference, optimized for your Mac
https://github.com/jundot/omlx
Benchmark Model: Qwen3.6-35B-A3B-Uncensored-Genesis-Hermes-V7-dequantized-oQ8e-mtp
Engine: Auto
Context: Code (Python)
================================================================================

Single Request Results
--------------------------------------------------------------------------------
Test                                TTFT(ms)    TPOT(ms)        pp TPS        tg TPS      E2E(s)    Throughput    Peak Mem
pp1024/tg128                           593.2       11.45  1726.2 tok/s    88.0 tok/s       2.055   560.7 tok/s    35.47 GB
pp4096/tg128                          2304.0       11.72  1777.8 tok/s    86.0 tok/s       3.776  1118.0 tok/s    36.35 GB
pp8192/tg128                          4822.6       14.61  1698.7 tok/s    71.2 tok/s       5.207  1578.3 tok/s    36.80 GB
pp16384/tg128                        10594.2       12.62  1546.5 tok/s    79.9 tok/s      12.205  1352.9 tok/s    37.69 GB
pp32768/tg128                        24743.9       13.66  1324.3 tok/s    73.8 tok/s      26.488  1241.9 tok/s    39.49 GB
pp65536/tg128                        63873.6       15.89  1026.0 tok/s    63.4 tok/s      65.901   996.4 tok/s    43.07 GB
pp131072/tg128                      193797.5       19.84   676.3 tok/s    50.8 tok/s     196.330   668.3 tok/s    50.36 GB
pp200000/tg128                      408283.1       24.26   489.9 tok/s    41.5 tok/s     411.381   486.5 tok/s    58.03 GB

Continuous Batching
pp1024 / tg128
--------------------------------------------------------------------------------
Batch           tg TPS   Speedup        pp TPS    pp TPS/req    TTFT(ms)      E2E(s)
1x          88.0 tok/s     1.00x  1726.2 tok/s  1726.2 tok/s       593.2       2.055
2x         147.4 tok/s     1.68x   654.8 tok/s   327.4 tok/s      3127.6       4.865
4x         221.4 tok/s     2.52x  1492.9 tok/s   373.2 tok/s      2631.8       5.056
8x         303.3 tok/s     3.45x  1477.5 tok/s   184.7 tok/s      5172.4       8.920

Intelligence Benchmark Comparison

               Mode    Sampled         Qwen3.6-35B-A3B-Uncensored-Genesis-Hermes-V7-dequantized-oQ8e-mtp
--------------------------------------------------------------------------------------------------------
MMLU           Sample  1000/14042                                                                  88.3%
MMLU_PRO       Sample  300/12032                                                                   81.3%
TRUTHFULQA     Full    817                                                                         81.8%
HUMANEVAL      Full    164                                                                         95.7%
LIVECODEBENCH  Sample  300/1055                                                                    39.0%

--- Detail ---

Model: Qwen3.6-35B-A3B-Uncensored-Genesis-Hermes-V7-dequantized-oQ8e-mtp
Benchmark         Accuracy   Correct   Total   Time(s)   Think
--------------------------------------------------------------
MMLU                 88.3%       883    1000    4470.9     Yes
MMLU_PRO             81.3%       244     300    2818.9     Yes
TRUTHFULQA           81.8%       668     817      3162     Yes
HUMANEVAL            95.7%       157     164    1495.8     Yes
LIVECODEBENCH        39.0%       117     300     14379     Yes