This is a Supervised PEFT(Parameter Efficient Fine-Tuning) based tuning of the Llama model of base conversational type to a code-based chatbot using the alpaca Dataset and SFT Trainer.

The model was trained under one epoch using SFT Trainer for up to 200 Steps by observing through significant gradient loss value (step-wise).

{ "num_train_epochs": 1, "fp16": false, "bf16": false, "per_device_train_batch_size": 4, "per_device_eval_batch_size": 4, "gradient_accumulation_steps": 4, "gradient_checkpointing": true, "max_grad_norm": 0.3, "learning_rate": 2e-4, "weight_decay": 0.001, "optim": "paged_adamw_32bit", "lr_scheduler_type": "cosine", "max_steps": -1, "warmup_ratio": 0.03, "group_by_length": true, "save_steps": 0, "logging_steps": 25, "base_lrs": [0.0002, 0.0002], "last_epoch": 199, "verbose": false, "_step_count": 200, "_get_lr_called_within_step": false, "_last_lr": [0.00019143163189119916, 0.00019143163189119916], "lr_lambdas": [{}, {}] }

These configurations (trained weights) are injected into the base model using PeftModel.from_pretrained() method.

Refer to this Github repo for notebooks: https://github.com/mr-nobody15/codebot_llama/tree/main