You can deploy this model and start earning money today!

islam-hajosman/
llama3_instruct_fine_tuned_bahn_1k_lora_adapter

Name: llama3_instruct_fine_tuned_bahn_1k_lora_adapter
Availability: InStock

peft

safetensors

islam-hajosman/deutsche_bahn_faq_1k

meta-llama/Meta-Llama-3-8B-Instruct

adapter

apache-2.0

Views

No views yet

Model Card for LoRa Adaptor: Llama 3 Instruct Fine-Tuned on Deutsche Bahn FAQ

Model Overview

Model Name: islam-hajosman/llama3_instruct_fine_tuned_bahn_1k_v1_lora_adapter
Architecture: Llama 3 Instruct with LoRa Adaptor
Quantization: 4-bit NF4 with double quantization
Domain-Specific Fine-Tuning Dataset: islam-hajosman/deutsche_bahn_faq_1k

This model card describes the LoRa adaptor fine-tuned to improve responses to FAQs from the Deutsche Bahn website. This project is part of a Master's thesis aiming to enhance domain-specific performance.

Fine-Tuning Configuration

LoRA Configuration

python

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.0,
    bias="none",
    target_modules=['q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj'],
    task_type="CAUSAL_LM"
)
model = get_peft_model(model, lora_config)

Hardware Used

GPU: 1x H100 (80 GB PCIe)
CPU: 26 cores
RAM: 205.4 GB
Storage: 1.1 TB SSD
Cost: $2.5 per hour

Training Summary

Total Trainable Parameters: 0.915% of 8B parameters
LoRA-Adaptor Size: 4.37GB
Training Time and Cost: $2 for 50 minutes
Number of Steps per Epoch: 16 (based on 1024 samples, batch size 8, gradient accumulation 8)

Performance Metrics

Training Completed:
- TrainOutput(global_step=480, training_loss=0.28411184588912874, metrics={'train_runtime': 3012.7974, 'train_samples_per_second': 10.197, 'train_steps_per_second': 0.159, 'total_flos': 3.871795189898281e+17, 'train_loss': 0.28411184588912874, 'epoch': 30.0})

Weights & Biases Tracking

Weights & Biases Dashboard

Usage

To use this LoRa adaptor model, load it from Huggingface using the model name islam-hajosman/llama3_instruct_fine_tuned_bahn_1k_v1_lora_adapter. This model is optimized for providing domain-specific answers to Deutsche Bahn FAQ.