GPT-2 Fine-Tuned on Dolly 15K for Instruction Following

1. Overview

• Follow natural language instructions
• Provide helpful and coherent answers
• Handle a variety of tasks such as question answering, classification, and explanation

2. Dataset

• 15,000 high-quality, human-written instruction–response pairs
• Covers multiple task types:
  • Open-ended question answering
  • Summarization
  • Classification and sentiment analysis
  • Brainstorming and creative tasks
  • Role-playing instructions

3. Training Setup

• Epochs: 3
• Max sequence length: 512
• Optimizer: AdamW (via Transformers Trainer)
• Learning rate: 5e-5
• Batch size: 4 per device with gradient_accumulation_steps = 4
• Weight decay: 0.01
• Warmup ratio: 0.03
• Hardware: single GPU (for example, Colab or similar environment)

• GPT-2 does not have a dedicated pad token.
• The tokenizer uses the EOS token as the pad token.
• pad_token_id is set to eos_token_id so that batched training works correctly.

4. Intended Use

• Small-scale instruction-following experiments
• Educational demonstrations of fine-tuning on instruction data
• Prototyping simple assistants for constrained domains
• Research on alignment and instruction tuning of smaller models

• High-stakes decision-making
• Production systems without additional safety mechanisms
• Domains requiring guaranteed factual accuracy or reliability

5. How to Use

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "Easonwangzk/gpt2-dolly-15k-instruct"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name).to(
    "cuda" if torch.cuda.is_available() else "cpu"
)

def generate_response(instruction, max_new_tokens=200):
    prompt = f"Instruction:\\instruction\\n\\nResponse:\\n"
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(
        **inputs,
        max_new_tokens=max_new_tokens,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        pad_token_id=tokenizer.eos_token_id,
    )
        return tokenizer.decode(outputs[0], skip_special_tokens=True)

instruction = "List 3 use cases for blockchain."
print(generate_response(instruction))

Instruction:
<your instruction here>

Response:

Instruction:
<your instruction here>

Context:
<relevant background or input>

Response:

6. Example Prompts

1. List 3 use cases for blockchain.
2. Explain photosynthesis.
3. Classify the following sentence as Positive or Negative sentiment: "I loved the customer service!"
4. Why is the sky blue?
5. Pretend you are a travel agent. Recommend a weekend getaway in the US for someone who loves hiking.
6. Compare the benefits of solar versus wind energy.

• Produces more direct and instruction-aligned answers
• Provides clearer and more structured explanations
• Avoids drifting into unrelated text as often

7. Limitations and Risks

• The model is based on GPT-2 without reinforcement learning from human feedback.
• It can hallucinate facts or produce incorrect information.
• It may generate biased or inappropriate content if prompted in certain ways.
• It does not have any built-in safety filters or content moderation.

• Manually review outputs before using them in any critical context.
• Avoid using this model in settings where mistakes could cause harm.

8. High-Level Training Procedure

1. Load Dolly 15K via the datasets library.
2. Convert each record to a unified instructional prompt format (Instruction, optional Context, and Response).
3. Tokenize with the GPT-2 tokenizer, using the EOS token as padding.
4. Fine-tune GPT-2 using the Hugging Face Trainer for several epochs.
5. Save the fine-tuned model and tokenizer, then push them to the Hugging Face Hub.

9. Acknowledgements

• Databricks Dolly 15K dataset
• GPT-2 and the Transformers library by Hugging Face