Description
This language model is the version 0.0 of a Gradio Coding Assistant. It is an instruction fine-tuned version of
StarCoder that is
designed to provide assistance to developers who use
gradio.
Dataset
The dataset is multi-source. Its content comes from the following sources
More precisely, we looked into
the-stack-dedup which contain codes permissive licenses. We shortlisted the files whose
content incorporated the keyword
gradio.
We scrapped all the issues of the official repository
the-gradio-app/gradio and added them to our training dataset.
- Spaces on Hugging Face Hub
We used the
HuggingFace_Hub API to scrape the data from the spaces which are designed with gradio. We kept track of those
with permissive licenses, namely MIT and Apache 2.0. This set of code was further deduplicated.
Training setting and hyperparameters
For our fine-tuning, we decided to follow a 2-step strategy.
- Pretraining (Fine-tuning) with next token prediction on the previously built gradio dataset (this step should familiarize the model with the gradio syntax.).
- Instruction fine-tuning on an instruction dataset (this step should make the model conversational.).
For both steps, we made use of parameter-efficient fine-tuning via the library PEFT, more precisely LoRA. Our
training script is the famous starcoder fine-tuning script.
Resources
Our training was done of 8 A100 GPUs of 80GB.
Pretraining
These are the parameters that we used :
- learning rate : 5e-4
- warmup_steps :
- gradient_accumulation_steps : 4
- batch_size : 1
- sequence length : 2048
- max_steps : 1000
- warmup_steps : 5
- weight_decay : 0.05
- learning rate scheduler : cosine
LORA PARAMETERS :
- r = 16
- alpha = 32
- dropout = 0.05
We stopped the training before the end and kept the checkpoint-100 for the second step.
Fine-tuning
This step consisted into the instruction fine-tuning of the previous checkpoint. For that purpose, we used a modified version of
openassistant-guanaco.
The template for the instruction fine-tuning was
Question: {question}\n\nAnswer: {answer}. We used exactly the same parameters we used during the pretraining and we kept the
checkpoint-50.
Usage
The usage is straightforward and very similar to any other instruction fine-tuned model.
1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3checkpoint_name="ArmelR/starcoder-gradio-v0"
4model = AutoModelForCausalLM.from_pretrained(checkpoint_name)
5tokenizer = AutoTokenizer.from_pretrained(checkpoint_name)
6
7prompt = "Create a gradio application that help to convert temperature in celcius into temperature in Fahrenheit"
8inputs = tokenizer(f"Question: {prompt}\n\nAnswer: ", return_tensors="pt")
9
10outputs = model.generate(
11 inputs["input_ids"],
12 temperature=0.2,
13 top_p=0.95,
14 max_new_tokens=200
15)
16
17input_len=len(inputs["input_ids"])
18print(tokenizer.decode(outputs[0][input_len:]))
Updates
Gradio dataset .filter(lambda x : ("gradio" in x["content"] or "gr." in x["content"]) and "streamlit" not in x["content"])
Guanaco ArmelR/oasst1_guanaco
- StarCoderbase (950, 1350)
- max_steps = 2000
- shuffle_buffer = 100
- batch_size = 2
- gradient_accumulation_steps = 4
- num_warmup_steps = 100
- weight_decay = 0.01
- StarCoderplus (2000)
Guanaco multi-turn (HuggingFaceH4/oasst1_en)
More information
For further information, refer to
StarCoder.
