Views
No views yet
1import torch
2from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig
3
4model_name = "xinchen9/Llama3.1_CoT"
5tokenizer = AutoTokenizer.from_pretrained(model_name)
6model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, device_map="auto")
7model.generation_config = GenerationConfig.from_pretrained(model_name)
8model.generation_config.pad_token_id = model.generation_config.eos_token_id| Metric | Value |
|---|---|
| Avg. | 13.35 |
| IFEval (0-Shot) | 22.46 |
| BBH (3-Shot) | 19.90 |
| MATH Lvl 5 (4-Shot) | 1.51 |
| GPQA (0-shot) | 5.15 |
| MuSR (0-shot) | 11.77 |
| MMLU-PRO (5-shot) | 19.32 |