This is a Llama-2-7b-hf model fine-tuned using QLoRA (4-bit precision) on the mlabonne/alpagasus dataset, which is a high-quality subset (9k samples) of the Alpaca dataset (52k samples).

🔧 Training

It was trained on an RTX 3090 using the 🐜🔧TinyTuner. Parameters:

yaml

# Dataset
dataset_name: mlabonne/alpagasus
prompt_template: alpaca
max_seq_length: 512
val_set_size: 0.01

# Loading
load_in_8bit: false
load_in_4bit: true
bf16: true
fp16: false
tf32: true

# Lora
adapter: qlora
lora_model_dir:
lora_r: 8
lora_alpha: 16
lora_dropout: 0.1
lora_target_modules:
  - q_proj
  - v_proj
lora_fan_in_fan_out:

# Training
learning_rate: 0.00002
micro_batch_size: 24
gradient_accumulation_steps: 1
num_epochs: 3
lr_scheduler_type: cosine
optim: paged_adamw_32bit
group_by_length: true
warmup_ratio: 0.03
eval_steps: 0.01
save_strategy: epoch
logging_steps: 1
weight_decay: 0
max_grad_norm:
max_steps: -1
gradient_checkpointing: true

# QLoRA
bnb_4bit_compute_dtype: float16
bnb_4bit_quant_type: nf4
bnb_4bit_use_double_quant: false

💻 Usage

python

# pip install transformers accelerate

from transformers import AutoTokenizer
import transformers
import torch

model = "mlabonne/alpagasus-2-7b"
prompt = "What is a large language model?"

tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    torch_dtype=torch.float16,
    device_map="auto",
)

sequences = pipeline(
    f'### Instruction: {prompt}',
    do_sample=True,
    top_k=10,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
    max_length=200,
)
for seq in sequences:
    print(f"Result: {seq['generated_text']}")