vllm (pretrained=/root/autodl-tmp/output8575,add_bos_token=true,tensor_parallel_size=2,max_model_len=2048,dtype=bfloat16), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto
| Tasks | Version | Filter | n-shot | Metric | | Value | | Stderr |
|---|
| gsm8k | 3 | flexible-extract | 5 | exact_match | ↑ | 0.796 | ± | 0.0255 |
| | strict-match | 5 | exact_match | ↑ | 0.780 | ± | 0.0263 |
vllm (pretrained=/root/autodl-tmp/mistral-nemo-gutenberg-12B-v4,add_bos_token=true,tensor_parallel_size=2,max_model_len=2048,dtype=bfloat16), gen_kwargs: (None), limit: 250.0, num_fewshot: 5, batch_size: auto
| Tasks | Version | Filter | n-shot | Metric | | Value | | Stderr |
|---|
| gsm8k | 3 | flexible-extract | 5 | exact_match | ↑ | 0.804 | ± | 0.0252 |
| | strict-match | 5 | exact_match | ↑ | 0.788 | ± | 0.0259 |
