Note: Results are for reference only and may vary depending on hardware, software configuration, and workload.
| Test | TTFT(ms) | TPOT(ms) | pp TPS | tg TPS | E2E(s) | Throughput | Peak Mem |
|---|---|---|---|---|---|---|---|
| pp1024/tg128 | 1065.4 | 23.12 | 961.1 tok/s | 43.6 tok/s | 4.012 | 287.1 tok/s | 11.44 GB |
| pp4096/tg128 | 4695.0 | 24.52 | 872.4 tok/s | 41.1 tok/s | 7.836 | 539.0 tok/s | 11.85 GB |
| Batch | tg TPS | Speedup | pp TPS | pp TPS/req | TTFT(ms) | E2E(s) |
|---|---|---|---|---|---|---|
| 1x | 43.6 tok/s | 1.00x | 961.1 tok/s | 961.1 tok/s | 1065.4 | 4.012 |
| 2x | 50.7 tok/s | 1.16x | 785.2 tok/s | 392.6 tok/s | 2608.1 | 7.659 |
| 4x | 55.6 tok/s | 1.28x | 676.9 tok/s | 169.2 tok/s | 5856.2 | 15.256 |
Note: Each benchmark round tests only 30 questions. Results are for reference only.
| Benchmark | Accuracy | Correct | Total | Time(s) | Think |
|---|---|---|---|---|---|
| MMLU | 63.3% | 19 | 30 | 32 | No |
| TRUTHFULQA | 83.3% | 25 | 30 | 15.2 | No |
| GSM8K | 93.3% | 28 | 30 | 188.9 | No |
| MATHQA | 13.3% | 4 | 30 | 98.5 | No |
| HUMANEVAL | 93.3% | 28 | 30 | 212.4 | No |
