Views
No views yet
learning_rate=2e-4
lora_r=64
lora_alpha=16
model_max_length=8192### Instruction:
<instruction>
### Response:1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3model_name_or_path="uukuguy/speechless-thoughts-mistral-7b-v1.0"
4tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, trust_remote_code=True)
5 model = AutoModelForCausalLM.from_pretrained(model_name_or_path, device_map="auto", trust_remote_code=True).eval()
6
7 system = ""Below is an instruction that describes a task.\nWrite a response that appropriately completes the request.\n\n""
8 prompt = f"{system}\n\n### Instruction:\n{instruction}\n\n### Response:"
9
10 inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
11 pred = model.generate(**inputs, max_length=4096, do_sample=True, top_k=50, top_p=0.99, temperature=0.9, num_return_sequences=1)
12 print(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))| Metric | Value |
|---|---|
| humaneval-python |
1{'ARC (acc_norm)': ,
2'HellaSwag (acc_norm)': ,
3'MMLU (acc)': ,
4'TruthfulQA (mc2)': ,
5'Winoground (acc)': ,
6'GSM8K (acc)': ,
7'DROP (f1)': ,
8'Open LLM Score': }| Metric | Value |
|---|---|
| Avg. | 59.36 |
| ARC (25-shot) | 58.53 |
| HellaSwag (10-shot) | 81.25 |
| MMLU (5-shot) | 54.59 |
| TruthfulQA (0-shot) | 48.09 |
| Winogrande (5-shot) | 78.14 |
| GSM8K (5-shot) | 35.18 |