from transformers import pipeline
model_name = "MadMarx37/deepseek-coder-1.3b-python-peft"
def generate_output(input):
# Run text generation pipeline with our next model
pipe = pipeline(task="text-generation", model=model_name, tokenizer=model_name, max_length=max_length)
result = pipe(input)
print(result[0]['generated_text'])
Training Details
Training Hyperparameters
Training regime: fp16 mixed-precision with original model loaded in 4bits with bitsandbytes
learning_rate = 2e-3
lr_scheduler_type = 'cosine_with_restarts'
max_grad_norm = 0.001
weight_decay = 0.001
num_train_epochs = 15
eval_strategy = "steps"
eval_steps = 25
Speeds, Sizes, Times [optional]
1.3B parameters. Training time of ~2 hours on an RTX3080.
Standard training and eval loss from the HF SFTTrainer.
Results
Training Loss: 0.074100
Validation Loss: 0.022271
Summary
The training had some unstability in the gradient norms, but the overall trend in both training and validation loss
were downward, and validation loss has almost plateaud, which is ideally where we want our model. The code generation on the same
prompts that we tested the original model on also seem better with the finetuned model. A good way to make the model better, if
we wanted to increase the finetuning data, would be to also increase the epochs.
