vllm (pretrained=/root/autodl-tmp/magnum-v4-12b-W8A8,add_bos_token=true,tensor_parallel_size=2,max_model_len=2048,dtype=bfloat16), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto
| Tasks | Version | Filter | n-shot | Metric | | Value | | Stderr |
|---|
| gsm8k | 3 | flexible-extract | 5 | exact_match | ↑ | 0.776 | ± | 0.0264 |
| | strict-match | 5 | exact_match | ↑ | 0.736 | ± | 0.0279 |
vllm (pretrained=/root/autodl-tmp/magnum-v4-12b-W8A8,add_bos_token=true,tensor_parallel_size=2,max_model_len=2048,dtype=float16), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto
| Tasks | Version | Filter | n-shot | Metric | | Value | | Stderr |
|---|
| gsm8k | 3 | flexible-extract | 5 | exact_match | ↑ | 0.768 | ± | 0.0268 |
| | strict-match | 5 | exact_match | ↑ | 0.744 | ± | 0.0277 |
vllm (pretrained=/root/autodl-tmp/magnum-v4-12b,add_bos_token=true,tensor_parallel_size=2,max_model_len=2048,dtype=bfloat16), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto
| Tasks | Version | Filter | n-shot | Metric | | Value | | Stderr |
|---|
| gsm8k | 3 | flexible-extract | 5 | exact_match | ↑ | 0.784 | ± | 0.0261 |
| | strict-match | 5 | exact_match | ↑ | 0.752 | ± | 0.0274 |
vllm (pretrained=/root/autodl-tmp/magnum-v4-12b,add_bos_token=true,tensor_parallel_size=2,max_model_len=2048,dtype=float16), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto
| Tasks | Version | Filter | n-shot | Metric | | Value | | Stderr |
|---|
| gsm8k | 3 | flexible-extract | 5 | exact_match | ↑ | 0.776 | ± | 0.0264 |
| | strict-match | 5 | exact_match | ↑ | 0.748 | ± | 0.0275 |
