Views
No views yet
VLLM_USE_V1=1 VLLM_ATTENTION_BACKEND="FLASH_ATTN" lm_eval --model vllm --model_args pretrained="./Valkyrie-49B-v1-W4A16-G64/",add_bos_token=true,trust_remote_code=True,max_model_len=16384,gpu_memory_utilization=0.6 --tasks gsm8k,piqa,hellaswag,rte,mbpp,humaneval --batch_size 'auto' --confirm_run_unsafe_code| Tasks |Version| Filter |n-shot| Metric | |Value | |Stderr|
|---------|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k | 3|flexible-extract| 5|exact_match|↑ |0.9075|± |0.0080|
| | |strict-match | 5|exact_match|↑ |0.9075|± |0.0080|
|hellaswag| 1|none | 0|acc |↑ |0.6465|± |0.0048|
| | |none | 0|acc_norm |↑ |0.8416|± |0.0036|
|humaneval| 1|create_test | 0|pass@1 | |0.4451|± |0.0389|
|mbpp | 1|none | 3|pass_at_1 |↑ |0.6800|± |0.0209|
|piqa | 1|none | 0|acc |↑ |0.8248|± |0.0089|
| | |none | 0|acc_norm |↑ |0.8335|± |0.0087|
|rte | 1|none | 0|acc |↑ |0.7545|± |0.0259|| Tasks | Version | Filter | n-shot | Metric | Value | Stderr | ||
|---|---|---|---|---|---|---|---|---|
| gsm8k | 3 | flexible-extract | 5 | exact_match | ↑ | 0.9242 | ± | 0.0073 |
| strict-match | 5 | exact_match | ↑ | 0.9219 | ± | 0.0074 | ||
| hellaswag | 1 | none | 0 | acc | ↑ | 0.6548 | ± | 0.0047 |
| none | 0 | acc_norm | ↑ | 0.8496 | ± | 0.0036 | ||
| humaneval | 1 | create_test | 0 | pass@1 | 0.4329 | ± | 0.0388 | |
| mbpp | 1 | none | 3 | pass_at_1 | ↑ | 0.6800 | ± | 0.0209 |
| piqa | 1 | none | 0 | acc | ↑ | 0.8270 | ± | 0.0088 |
| none | 0 | acc_norm | ↑ | 0.8400 | ± | 0.0086 | ||
| rte | 1 | none | 0 | acc | ↑ | 0.7220 | ± | 0.0270 |
| Task | Metric | Base Score | Quant Score | Delta |
|---|---|---|---|---|
| GSM8K | exact_match | 0.9242 | 0.9075 | -1.67% |
| HellaSwag | acc | 0.6548 | 0.6465 | -0.83% |
| acc_norm | 0.8496 | 0.8416 | -0.80% | |
| HumanEval | pass@1 | 0.4329 | 0.4451 | +1.22% |
| MBPP | pass@1 | 0.6800 | 0.6800 | 0.00% |
| PIQA | acc | 0.8270 | 0.8248 | -0.22% |
| acc_norm | 0.8400 | 0.8335 | -0.65% | |
| RTE | acc | 0.7220 | 0.7545 | +3.25% |