Hardware: M5 MacBook Air 32GB
Inference Framework: oMLX v0.5.3
Max Concurrent Requests: 4
Settings:
- Thinking: Disabled
- Chat template parameter: enable_thinking=false (forced)
- TurboQuant KV Cache: Enabled (disable before oMLX v0.5.2.dev1)
- Lightning MTP: Enabled (key speed improvement)

Performance Benchmarks

Note: Results are for reference only and may vary depending on hardware, software configuration, and workload.

Test	TTFT(ms)	TPOT(ms)	pp TPS	tg TPS	E2E(s)	Throughput	Peak Mem
pp1024/tg128	1052.3	17.53	973.1 tok/s	57.5 tok/s	3.288	350.4 tok/s	13.09 GB
pp4096/tg128	3634.2	19.08	1127.1 tok/s	52.8 tok/s	6.078	695.0 tok/s	13.80 GB

Continuous Batching (pp1024 / tg128)

Batch	tg TPS	Speedup	pp TPS	pp TPS/req	TTFT(ms)	E2E(s)
1x	57.5 tok/s	1.00x	973.1 tok/s	973.1 tok/s	1052.3	3.288
2x	67.3 tok/s	1.17x	882.8 tok/s	441.4 tok/s	2319.8	6.124
4x	101.3 tok/s	1.76x	867.5 tok/s	216.9 tok/s	4580.2	9.776

Intelligence Benchmark

Note: Each benchmark round tests only 30 questions. Results are for reference only.

Benchmark	Accuracy	Correct	Total	Time(s)	Think
MMLU	70.0%	21	30	43.5	No
TRUTHFULQA	90.0%	27	30	15.9	No
GSM8K	90.0%	27	30	125	No
MATHQA	36.7%	11	30	45.2	No
HUMANEVAL	80.0%	24	30	191.1	No