Views
No views yet
Astraios-FFT is an instruction tuned model with 15.5B parameters created by finetuning StarCoderBase on CommitPackFT & OASST as described in the Astraios paper.
| Data | CommitPackFT+OASST | Filtered version of CommitPack and OASST for high-quality commit messages that resemble instructions |
|---|---|---|
| Model | Astraios-1B | Collection of StarCoderBase-1B models instruction tuned on CommitPackFT + OASST with different tuning methods |
| Astraios-3B | Collection of StarCoderBase-3B (3B parameters) models instruction tuned on CommitPackFT + OASST with different tuning methods | |
| Astraios-7B | Collection of StarCoderBase-7B (7B parameters) models instruction tuned on CommitPackFT + OASST with different tuning methods | |
| Astraios-16B | Collection of StarCoderBase-16B (16B parameters) models instruction tuned on CommitPackFT + OASST with different tuning methods | |
| Evaluation | BigCloneBench | Dataset for clone detection; We use 2,000 samples for evaluation |
| Devign | Dataset for defect detection; We use 2,000 samples for evaluation | |
| HumanEvalPack | Extension of OpenAI's HumanEval to cover 3 scenarios across 6 languages | |
| ReCode | Dataset for the robustness of code generation, covering 4 variants | |
| Asleep At The Keyboard | Datasets for security of code generation; We use DoW for evaluation |
1# pip install -q transformers
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4checkpoint = "bigcode/astraios-fft"
5model = AutoModelForCausalLM.from_pretrained(checkpoint)
6device = "cuda" # for GPU usage or "cpu" for CPU usage
7
8tokenizer = AutoTokenizer.from_pretrained(checkpoint)
9model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)
10
11inputs = tokenizer.encode("Question: Please write a function in Python that performs bubble sort.
12
13Answer:", return_tensors="pt").to(device)
14outputs = model.generate(inputs)
15print(tokenizer.decode(outputs[0]))