Views
No views yet
### Instruction:
<instruction>
### Response:1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3model_name_or_path="uukuguy/speechless-thoughts-mistral-7b"
4tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, trust_remote_code=True)
5 model = AutoModelForCausalLM.from_pretrained(model_name_or_path, device_map="auto", trust_remote_code=True).eval()
6
7 system = ""Below is an instruction that describes a task.\nWrite a response that appropriately completes the request.\n\n""
8 prompt = f"{system}\n\n### Instruction:\n{instruction}\n\n### Response:"
9
10 inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
11 pred = model.generate(**inputs, max_length=4096, do_sample=True, top_k=50, top_p=0.99, temperature=0.9, num_return_sequences=1)
12 print(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))| Metric | Value |
|---|---|
| humaneval-python |
1{'ARC (acc_norm)': ,
2'HellaSwag (acc_norm)': ,
3'MMLU (acc)': ,
4'TruthfulQA (mc2)': ,
5'Winoground (acc)': ,
6'GSM8K (acc)': ,
7'DROP (f1)': ,
8'Open LLM Score': }| Metric | Value |
|---|---|
| Avg. | 59.72 |
| ARC (25-shot) | 58.96 |
| HellaSwag (10-shot) | 80.71 |
| MMLU (5-shot) | 60.11 |
| TruthfulQA (0-shot) | 49.91 |
| Winogrande (5-shot) | 77.82 |
| GSM8K (5-shot) | 30.78 |