Model Card for Model ID
Model Details
Model Description
This is a 🤗 transformers model fine-tuned from Meta's LLaMA2-7B base using the Alpaca-style instruction dataset and QLoRA (Quantized Low-Rank Adaptation) via the PEFT library.
The model has been instruction-tuned to follow human prompts in a conversational and task-completion setting.
- Developed by: RajTejani
- Model type: Causal Language Model (Decoder-only Transformer), fine-tuned with PEFT QLoRA
- Language(s) (NLP): English (en)
- License: LLaMA 2 Community License Agreement ([Meta AI]
- **Finetuned from model meta-llama/Llama-2-7b-hf
Direct Use
This model is intended to be used as an instruction-following assistant. It can respond to natural language prompts and complete tasks such as:
- Question answering
- Summarization
- Text generation and completion
- Simple reasoning tasks
- Writing assistance
Downstream Use
The model can be further fine-tuned on domain-specific datasets for specialized applications such as customer support bots, coding assistants, or educational tools.
Out-of-Scope Use
This model is not intended for:
- Generating harmful, abusive, or misleading content
- Making critical decisions in high-stakes domains (medical, legal, financial) without human oversight
- Any use that violates Meta's LLaMA 2 Community License Agreement
- Serving as a factual knowledge base — it may hallucinate and produce incorrect information
Bias, Risks, and Limitations
- The base model (LLaMA2-7B) was trained on large-scale internet data and may reflect societal biases present in that data.
- Fine-tuning on the Alpaca dataset, which is synthetically generated via GPT, may introduce additional artifacts or biases from the teacher model.
- The model can produce confident-sounding but factually incorrect outputs (hallucinations).
- It is a relatively small 7B parameter model and may underperform larger models on complex reasoning, multi-step tasks, or nuanced language understanding.
- The model is primarily optimized for English and may perform poorly on other languages.
Recommendations
Users (both direct and downstream) should be made aware of the risks, biases, and limitations of the model. It is strongly recommended to:
- Always verify critical information produced by the model with trusted sources.
- Apply safety filters or guardrails in production deployments.
- Avoid use in sensitive or high-risk decision-making scenarios without human oversight.
How to Get Started with the Model
Use the code below to get started with the model.
1import torch
2from transformers import AutoTokenizer, AutoModelForCausalLM
3from peft import PeftModel, PeftConfig
4
5# Load the PEFT config and base model
6peft_model_id = "RajTejani/llama2-7b-alpaca-qlora"
7config = PeftConfig.from_pretrained(peft_model_id)
8
9# Load base model in 4-bit quantization
10from transformers import BitsAndBytesConfig
11
12bnb_config = BitsAndBytesConfig(
13 load_in_4bit=True,
14 bnb_4bit_use_double_quant=True,
15 bnb_4bit_quant_type="nf4",
16 bnb_4bit_compute_dtype=torch.bfloat16,
17)
18
19base_model = AutoModelForCausalLM.from_pretrained(
20 config.base_model_name_or_path,
21 quantization_config=bnb_config,
22 device_map="auto",
23)
24
25tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)
26tokenizer.pad_token = tokenizer.eos_token
27
28# Load the fine-tuned PEFT adapter
29model = PeftModel.from_pretrained(base_model, peft_model_id)
30model.eval()
31
32# Inference with Alpaca-style prompt
33def generate_response(instruction, input_text=""):
34 if input_text:
35 prompt = f"### Instruction:\n{instruction}\n\n### Input:\n{input_text}\n\n### Response:\n"
36 else:
37 prompt = f"### Instruction:\n{instruction}\n\n### Response:\n"
38
39 inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
40 with torch.no_grad():
41 outputs = model.generate(
42 **inputs,
43 max_new_tokens=256,
44 temperature=0.7,
45 do_sample=True,
46 top_p=0.9,
47 )
48 return tokenizer.decode(outputs[0], skip_special_tokens=True)
49
50# Example usage
51response = generate_response("Explain what a transformer model is in simple terms.")
52print(response)
Training Details
Training Data
The model was fine-tuned on the Alpaca-QLoRA dataset, an instruction-following dataset in the Alpaca format. It consists of (instruction, input, output) triplets originally derived from Stanford Alpaca's 52K GPT-generated instructions.
- Dataset:
yahma/alpaca-cleaned (or the original tatsu-lab/alpaca)
- Size: ~52,000 instruction-following examples
- Language: English
- Format: Alpaca prompt template (
### Instruction, ### Input, ### Response)
Training Procedure
Preprocessing [optional]
- Tokenization was performed using the LLaMA2 tokenizer.
- Prompts were formatted using the standard Alpaca instruction template.
- Sequences were truncated and padded to a maximum length of 512 tokens.
- The
pad_token was set to eos_token.
Training Hyperparameters
| Hyperparameter | Value |
|---|
| Training regime | 4-bit QLoRA (NF4 quantization, bfloat16 compute) |
| LoRA rank (r) | 64 |
| LoRA alpha | 16 |
| LoRA dropout | 0.1 |
| Target modules | q_proj, v_proj |
| Learning rate | 2e-4 |
| LR scheduler | Cosine |
| Batch size | 4 (per device) |
| Gradient accumulation steps | 4 |
| Effective batch size | 16 |
| Epochs | 3 |
| Warmup steps | 100 |
| Optimizer | paged_adamw_32bit |
| Max sequence length | 512 |
| Weight decay | 0.001 |
Speeds, Sizes, Times [optional]
- Trainable parameters: ~4M (LoRA adapter only, ~0.06% of total parameters)
- Total model parameters: ~7B (frozen base) + ~4M (trainable LoRA)
- Training time: [More Information Needed]
- GPU used: [e.g., 1x NVIDIA A100 40GB / RTX 3090]
Evaluation
Testing Data, Factors & Metrics
Testing Data
[More Information Needed — e.g., a held-out split of the Alpaca dataset or a separate instruction-following benchmark such as MT-Bench or Vicuna Eval]]
Factors
Evaluation considered the following factors:
- Instruction-following accuracy
- Fluency and coherence of generated responses
- Factual correctness on knowledge-grounded prompts
Metrics
- Perplexity on held-out validation set
- ROUGE scores (ROUGE-1, ROUGE-2, ROUGE-L) for generative tasks
- Human evaluation [optional]: Response quality rated on a 1–5 scale
Results
| Metric | Score |
|---|
| Perplexity | [More Information Needed] |
| ROUGE-1 | [More Information Needed] |
| ROUGE-L | [More Information Needed] |
Summary
The model demonstrates solid instruction-following capability on common NLP tasks after QLoRA fine-tuning. The QLoRA approach enables efficient fine-tuning at reduced memory cost while maintaining competitive performance compared to full fine-tuning.
Model Examination [optional]
The LoRA adapters were applied to the query and value projection matrices (q_proj, v_proj) across all transformer layers. The base LLaMA2-7B weights were frozen throughout training, with only the low-rank adapter weights updated.
Environmental Impact
Carbon emissions can be estimated using the
Machine Learning Impact calculator presented in
Lacoste et al. (2019).
- Hardware Type: T4 GPU
- Cloud Provider: Google Colab
- Compute Region: us-east-1
Technical Specifications [optional]
Model Architecture and Objective
- Base Architecture: LLaMA 2 (Decoder-only Transformer, 7B parameters)
- Fine-tuning Method: QLoRA (Quantized Low-Rank Adaptation) via the PEFT library
- Quantization: 4-bit NormalFloat (NF4) with double quantization
- Objective: Causal Language Modeling (next-token prediction) on instruction-formatted data
Compute Infrastructure
- Software:
- Python 3.10+
- PyTorch 2.x
- 🤗 Transformers
- 🤗 PEFT
- 🤗 Datasets
- bitsandbytes
- trl (SFTTrainer)
Glossary [optional]
- QLoRA: Quantized Low-Rank Adaptation — a parameter-efficient fine-tuning method that quantizes the base model to 4-bit and trains only small low-rank adapter matrices.
- PEFT: Parameter-Efficient Fine-Tuning — a family of techniques that fine-tune only a small subset of model parameters.
- LoRA: Low-Rank Adaptation — adapts frozen pre-trained weights by injecting trainable rank-decomposition matrices.
- NF4: NormalFloat 4-bit — a 4-bit data type optimized for normally distributed weights.
- Alpaca: An instruction-following dataset/model format developed by Stanford, using GPT-generated
(instruction, input, output) triples.
More Information [optional]
