Finally, the model we started training with our blog post is ready 🎉
After fine-tuning on generated data, it beats Replit 3b, Stability Code 3b and many other models. It almost beats
StarCoder ten times the size!
Model
Size
HumanEval pass@1
HumanEval pass@10
DeciCoder-1b
1b
19.1%
Refact-1.6-fim
1.6b
32.0%
53.0%
StableCode
3b
20.2%
33.8%
ReplitCode v1
3b
21.9%
CodeGen2.5-multi
7b
28.4%
47.5%
CodeLlama
7b
33.5%
59.6%
StarCoder
15b
33.6%
Likely, it's the best model for practical use in your IDE for code completion because it's smart and fast!
You can start using it right now by downloading the
Refact plugin. You can host the model yourself, too, using the
open source docker container.
And it's multi-language (see MultiPL-HumanEval and other metrics below) and it works as a chat (see the section below).
It Works As a Chat
The primary application of this model is code completion (infill) in multiple programming languages.
But it works as a chat quite well.
HumanEval results using instruction following (chat) format, against models specialized for chat only:
Model
Size
pass@1
pass@10
Refact-1.6-fim
1.6b
38.4%
55.6%
StableCode-instruct
3b
26.9%
36.2%
OctoGeeX
6b
44.7%
CodeLlama-instruct
7b
34.8%
64.3%
CodeGen2.5-instruct
7b
36.2%
60.87
CodeLlama-instruct
13b
42.7%
71.6%
StarChat-β
15b
33.5%
OctoCoder
15b
46.2%
Example
Fill-in-the-middle uses special tokens to identify the prefix/middle/suffix part of the input and output:
python
1# pip install -q transformers2from transformers import AutoModelForCausalLM, AutoTokenizer
34checkpoint ="smallcloudai/Refact-1_6B-fim"5device ="cuda"# for GPU usage or "cpu" for CPU usage67tokenizer = AutoTokenizer.from_pretrained(checkpoint)8model = AutoModelForCausalLM.from_pretrained(checkpoint, trust_remote_code=True).to(device)910prompt ='<fim_prefix>def print_hello_world():\n """<fim_suffix>\n print("Hello world!")<fim_middle>'1112inputs = tokenizer.encode(prompt, return_tensors="pt").to(device)13outputs = model.generate(inputs, max_length=100, temperature=0.2)14print("-"*80)15print(tokenizer.decode(outputs[0]))
Chat Format
The same model works as chat (experimental).
python
1prompt_template ="<empty_output>SYSTEM {system}\n" \
2"<empty_output>USER {query}\n" \
3"<empty_output>ASSISTANT"4prompt = prompt_template.format(system="You are a programming assistant",5 query="How do I sort a list in Python?")
Architecture
As described in more detail in the blog post, we used:
We also used LiON, flash attention, early dropout. It's not that innovative that you can't run it, in fact you can -- see an example below.
Pretraining
For the base model, we used our own dataset that contains code with permissive licenses only, and open text datasets.
Filtering is the key to success of this model:
We only used text in English
Only topics related to computer science
Applied heavy deduplication
The text to code proportion was 50:50, model trained for 1.2T tokens.
We don't release the base model, because its Fill-in-the-Middle (FIM) capability likes to repeat itself too much, so
its practical use is limited. But if you still want it, write us a message on Discord.
Finetuning
We tested our hypothesis that chat data should boost base model performance in FIM and
regular left-to-right code completion. We found that just 15% of open
codeinstruction-following datasets,
that we filtered for quality, improves almost all metrics.
Additionally, to improve FIM, we observed common failure modes, and prepared a synthetic dataset based on
The Stack dedup v1.1 to address them.
There is a distribution shift between typical code on the internet, and the code you write in your IDE.
The former is likely finished, so the model tries to come up with a suggestion that makes the code complete.
You are likely to have half-written code as you work on it, there is no single addition that can repair it
fully.
In practice, model needs to have a tendency to stop after a couple of lines are added, and sometimes don't write
anything at all. We found that just giving it empty completions, single line completions, multiline
completions that end with a smaller text indent or at least a newline -- makes it much more usable. This data
was used as the rest 85% of the finetune dataset.
The final model is the result of several attempts to make it work as good as possible for code completion,
and to perform well on a wide range of metrics. The best attempt took 40B tokens.
Limitations and Bias
The Refact-1.6B model was trained on text in English. But it has seen a lot more languages in
code comments. Its performance on non-English languages is lower, for sure.
Model Stats
Architecture: LLAMA-like model with multi-query attention
Objectives Fill-in-the-Middle, Chat
Tokens context: 4096
Pretraining tokens: 1.2T
Finetuning tokens: 40B
Precision: bfloat16
GPUs 64 NVidia A5000
Training time 28 days
License
The model is licensed under the BigScience OpenRAIL-M v1 license agreement
Citation
If you are using this model, please give a link to this page.
