OpenCSG stands for Converged resources, Software refinement, and Generative LM. The 'C' represents Converged resources, indicating the integration and full utilization of hybrid resources. The 'S' stands for Software refinement, signifying software that is refined by large models. The 'G' represents Generative LM, which denotes widespread, inclusive, and democratized generative large models.
The vision of OpenCSG is to empower every industry, every company, and every individual to own their models. We adhere to the principles of openness and open source, making the large model software stack of OpenCSG available to the community. We welcome everyone to use, send feedback, and contribute collaboratively.
Model Description
The StarCoder models are 15.5B parameter models trained on 80+ programming languages from The Stack (v1.2), with opt-out requests excluded.
Based on StarCoder2, opencsg-starcoder2-3b-v0.1 was fintuned by OpenCSG LLM Research Team througth full-paramters fine-tuning method.
Model Eval
HumanEval is the most common code generation benchmark for evaluating model performance, especially on the compeltion of code exercise cases.
Model evaluation is, to some extent, a metaphysics. Different models have different sensitivities to decoding methods, parameters and instructions.
It is impratical for us to manually set specific configurations for each fine-tuned model, because a real LLM should master general capabilities despite the parameters being manipulated by users.
Therefore, OpenCSG racked their brains to provide a relatively fair method to compare the fine-tuned models on the HumanEval benchmark.
To simplify the comparison, we chosed the Pass@1 metric for the Python language, but our fine-tuning dataset includes samples in multiple languages.
For fairness, we evaluated the original and fine-tuned StarCoder models based only on the prompts from the original cases, without including any other instructions.
Otherwise, we use the greedy decoding method for each model during evaluation.
Model
HumanEval python pass@1
starcoder
35.98%
opencsg-starcoder-v0.1
42.68%
starcoder2-3b
32.93%
opencsg-starcoder2-3b-v0.1
45.12%
starcoder2-7b
35.37%
opencsg-starcoder2-7b-v0.1
51.22%
starcoder2-15b
45.12%
opencsg-starcoder2-15b-v0.1
59.15%
TODO
We will provide more benchmark scores on fine-tuned models in the future.
We will provide different practical problems to evaluate the performance of fine-tuned models in the field of software engineering.
Model Usage
python
1from transformers import AutoTokenizer
2import transformers
3import torch
45model ="opencsg/opencsg-starcoder2-3b-v0.1"67tokenizer = AutoTokenizer.from_pretrained(model, trust_remote_code=True)8pipeline = transformers.pipeline(9"text-generation",10 model=model,11 torch_dtype=torch.float16,12 device_map="auto",13)14input_text ="""#Generate one test case for the following code.
15def quick_sort(arr):
16 if len(arr) < 2:
17 return arr
18 else:
19 pivot = arr[0]
20 less = [i for i in arr[1:] if i <= pivot]
21 greater = [i for i in arr[1:] if i > pivot]
22 return quick_sort(less) + [pivot] + quick_sort(greater)
23"""24sequences = pipeline(25 input_text,26 do_sample=False,27 top_k=10,28 temperature=0.1,29 top_p=0.95,30 num_return_sequences=1,31 eos_token_id=tokenizer.eos_token_id,32 max_length=256,33)34for seq in sequences:35print(seq['generated_text'][len(input_text):])
generate output
python
1# Test case2arr =[5,2,9,1,7]3print(quick_sort(arr))4
1from transformers import AutoTokenizer
2import transformers
3import torch
45model ="opencsg/opencsg-starcoder2-3b-v0.1"67tokenizer = AutoTokenizer.from_pretrained(model, trust_remote_code=True)8pipeline = transformers.pipeline(9"text-generation",10 model=model,11 torch_dtype=torch.float16,12 device_map="auto",13)14input_text ="""#Generate one test case for the following code.
15def quick_sort(arr):
16 if len(arr) < 2:
17 return arr
18 else:
19 pivot = arr[0]
20 less = [i for i in arr[1:] if i <= pivot]
21 greater = [i for i in arr[1:] if i > pivot]
22 return quick_sort(less) + [pivot] + quick_sort(greater)
23"""24sequences = pipeline(25 input_text,26 do_sample=False,27 top_k=10,28 temperature=0.1,29 top_p=0.95,30 num_return_sequences=1,31 eos_token_id=tokenizer.eos_token_id,32 max_length=256,33)34for seq in sequences:35print(seq['generated_text'][len(input_text):])
generate output
python
1# Test case2arr =[5,2,9,1,7]3print(quick_sort(arr))4
Additional thanks to @nicoboss for giving me access to his private supercomputer, enabling me to provide many more quants, at much higher speed, than I would otherwise be able to.
