This repository contains a LoRA fine-tuned adapter for CodeLlama-7b-Python-hf, trained to improve Python instruction-following and code generation.

Note:
This is a PEFT LoRA adapter, not a fully merged standalone model. You must load it on top of the base model.

Model Details

Base model: codellama/CodeLlama-7b-Python-hf
Fine-tuned for: Python instruction-following and code generation
Fine-tuning method: SFT + LoRA (PEFT)
Framework: Transformers + PEFT + TRL

Dataset Used

This adapter was fine-tuned on:

jtatman/python-code-dataset-500k
- Large-scale Python instruction → solution pairs
- Parquet format (~500k+ examples)

Training Configuration

LoRA Configuration

r: 32
lora_alpha: 16
Target modules:
q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

SFT Configuration

Epochs: 1
Learning rate: 2e-4
Scheduler: cosine
Warmup ratio: 0.03
Weight decay: 0.0
Train batch size: 4
Eval batch size: 4
Gradient accumulation steps: 16
Precision: bf16
Attention: flash_attention_2
Packing: enabled
Gradient checkpointing: enabled
Logging: every 50 steps + per epoch
Saving: per epoch (save_total_limit=2)

Evaluation Results

The model was evaluated using both language-modeling metrics and generation-quality metrics.

📉 Perplexity / Loss

Base model loss: 1.3214
Base model perplexity: 3.7486
Fine-tuned (LoRA) val/test loss: 0.7126
Fine-tuned (LoRA) val/test perplexity: 2.0394

📊 Generation Quality Metrics (Test)

Exact Match: 0.0033
Normalized Exact Match: 0.0033
BLEU: 18.43
chrF: 34.06
ROUGE-L (F1): 0.2417

🧠 CodeBERTScore (Mean)

Precision: 0.7187
Recall: 0.7724
F1: 0.7421
F3: 0.7657

🧾 Training Summary (from logs)

Train loss: ~0.6903
Eval loss: ~0.6877

Example Usage

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Base + adapter
base_id = "codellama/CodeLlama-7b-Python-hf"
adapter_id = "Tanneru/CodeLlama-7b-Python-hf-ft"

# Load tokenizer (repo includes tokenizer files)
tokenizer = AutoTokenizer.from_pretrained(adapter_id)

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    base_id,
    device_map="auto",
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
)

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

prompt = "Write a Python function that checks if a number is prime."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=256)

print(tokenizer.decode(out[0], skip_special_tokens=True))

Citation

If you use this model in your research or project, please cite it:

bibtex

@misc{tanneru2025codellamapythonft,
  title   = {CodeLlama-7b-Python-hf-ft},
  author  = {Tanneru},
  year    = {2025},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Tanneru/CodeLlama-7b-Python-hf-ft}}
}