StarCoder-1b-textbook is a finetuned version of
starcoderbase-1b on the
code_exercices dataset
It achieves 27.0 pass@1 on the
Human Eval coding benchmark while being only 1b parameters.
That is an improvement of almost 12 points over the starcoder 1b baseline, almost doubling the score.
The results (on the human eval benchmark) are on par with other open-source models like StarCoderBase (30.4) StarCoder(33.6) CodeGen-16B-Mono(29.3) while the model being 15 times smaller.
It still underperforms compared to other models like CodeLLama (53%) chat gpt 4 (82) or wizard coder (73.2), but these model are more than 30 times bigger.
1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3model = AutoModelForCausalLM.from_pretrained(
4 "jinaai/starcoder-1b-textbook", device_map='auto'
5 )
6
7tokenizer = AutoTokenizer.from_pretrained("jinaai/starcoder-1b-textbook")
8
9prompt = '''
10def unique(l: list):
11 """Return sorted unique elements in a list
12 >>> unique([5, 3, 5, 2, 3, 3, 9, 0, 123])
13 [0, 2, 3, 5, 9, 123]
14 """
15'''
16
17inputs = tokenizer(prompt.rstrip(), return_tensors="pt").to("cuda")
18
19generation_output = model.generate(
20 **inputs,
21 max_new_tokens=128,
22 eos_token_id=tokenizer.eos_token_id,
23 return_dict_in_generate=True,
24)
25
26s = generation_output.sequences[0]
27output = tokenizer.decode(s, skip_special_tokens=True)
28
29print(output)
We did full parameter fine-tuning and used a Nvidia a40 for 12 hours using a batch size of 128 and a micro-batch size of 8.
To reproduce the training just follow the training instructions in our
open source codebaseThis model was trained and released by
Jina.ai
