Views
No views yet
| EvalPlus | pass@1 |
|---|---|
| HumanEval | 86.6 |
| HumanEval+ | 83.5 |
| MBPP(v0.2.0) | 82.3 |
| MBPP+(v0.2.0) | 70.4 |
"Complete the following Python function:\n{prompt}"| Models | HumanEval | HumanEval+ |
|---|---|---|
| GPT-4-Turbo (April 2024) | 90.2 | 86.6 |
| GPT-4 (May 2023) | 88.4 | 81.17 |
| GPT-4-Turbo (Nov 2023) | 85.4 | 79.3 |
| CodeQwen1.5-7B-Chat | 83.5 | 78.7 |
| claude-3-opus (Mar 2024) | 82.9 | 76.8 |
| DeepSeek-Coder-33B-instruct | 81.1 | 75.0 |
| WizardCoder-33B-V1.1 | 79.9 | 73.2 |
| OpenCodeInterpreter-DS-33B | 79.3 | 73.8 |
| speechless-codellama-34B-v2.0 | 77.4 | 72 |
| GPT-3.5-Turbo (Nov 2023) | 76.8 | 70.7 |
| Llama3-70B-instruct | 76.2 | 70.7 |
apply_chat_template to show you how to load the tokenizer and model and how to generate contents. You should upgrade the transformers if you receive an error when loading the tokenizer1from transformers import AutoModelForCausalLM, AutoTokenizer
2device = "cuda" # the device to load the model onto
3
4model = AutoModelForCausalLM.from_pretrained(
5 "NTQAI/Nxcode-CQ-7B-orpo",
6 torch_dtype="auto",
7 device_map="auto"
8)
9tokenizer = AutoTokenizer.from_pretrained("NTQAI/Nxcode-CQ-7B-orpo")
10
11prompt = """Complete the following Python function:
12from typing import List
13
14
15def has_close_elements(numbers: List[float], threshold: float) -> bool:
16 """ Check if in given list of numbers, are any two numbers closer to each other than
17 given threshold.
18 >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
19 False
20 >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
21 True
22 """
23"""
24messages = [
25 {"role": "user", "content": prompt}
26]
27
28inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
29outputs = model.generate(inputs, max_new_tokens=512, do_sample=False, top_k=50, top_p=0.95, num_return_sequences=1, eos_token_id=tokenizer.eos_token_id)
30res = tokenizer.decode(outputs[0][len(inputs[0]):], skip_special_tokens=True)
31