A causal language model fine-tuned to generate conceptual explanations in the style of Richard Feynman, combining supervised warm-up with on-policy reinforcement learning (Group Relative Proximal Optimization, GRPO) and accelerated inference via vLLM. The model excels at breaking down complex ideas into first-principles intuitions, concrete analogies, and short, clear paragraphs.

Model Details

Model Description

This model builds on Qwen/Qwen2.5-1.5B-Instruct, with a multi-stage training pipeline:

Supervised Fine-Tuning (SFT)
- ~750 carefully designed prompt–completion pairs.
- Each pair follows a Feynman-style teaching pattern:
  - simple language,
  - layered explanation,
  - everyday analogies,
  - checks for understanding.
- Implemented with LoRA adapters for efficient parameter updates.
Reinforcement Learning with GRPO
- Reward Model: trained on GPT-4o-mini preference judgments between candidate completions.
- Objective: maximize adherence to “first-principle reasoning” and clarity of explanation.
- Algorithm: Group Relative Proximal Optimization (GRPO), a variant of PPO adapted for group-based baselines.
- On-policy rollouts were generated using the SFT-warm model, with iterative policy updates guided by the reward model.
Merging and Optimization
- LoRA adapters from SFT and RLHF stages were merged into the base model.
- Final model optimized for inference with vLLM, reducing latency and improving throughput.

Finetuned by: Khushal Mandavia
Model type: Decoder-only transformer (causal LM)
Base model: Qwen/Qwen2.5-1.5B-Instruct
Repository: huggingface.co/KhushalM/Qwen2.5-1.5-Feynman-GRPO-vLLM-Merged

Uses

Direct Use

Generate clear, analogy-driven explanations of technical or scientific topics.
Serve as an educational chatbot that teaches concepts step-by-step.

Downstream Use

Integrate into learning platforms for pedagogical clarity.
Generate lecture notes, study guides, or FAQs.

Out-of-Scope Use

Not intended for authoritative factual precision.
Not optimized for open-domain chit-chat or casual conversation.

Bias, Risks, and Limitations

Hallucinations: May invent plausible but incorrect details.
Oversimplification: Risk of leaving out nuance in pursuit of clarity.
Style Bias: Strongly favors the “Feynman teaching style,” which may be unsuitable for formal or technical contexts.

Recommendation: Fact-check outputs before use in professional or educational materials.

Training Procedure

Supervised Fine-Tuning

Dataset size: ~750 curated examples.
Framework: Hugging Face transformers with PEFT (LoRA).
Loss: Cross-entropy, optimized with AdamW.

Reward Model

Base: Qwen2.5-1.5B classifier head.
Training Data: ~3k pairwise preferences (generated via GPT-4o-mini comparisons).
Objective: Predict preference for completions that are simpler, analogy-rich, and structured.

GRPO (Reinforcement Learning Phase)

Algorithm: Group Relative Proximal Optimization (variant of PPO).
Rollouts: Generated with SFT-warm policy.
Group Baseline: Used relative ranking of completions within a rollout batch to stabilize training.
Reward: Scalar from reward model (0–1).
Stability tricks: Gradient clipping, KL penalty, early stopping.

Merge + Inference

LoRA adapters from SFT and GRPO merged.
Final model loaded into vLLM for inference optimization.
Latency reduced by ~2.5× compared to vanilla transformers inference.

Example Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

model_id = "KhushalM/Qwen2.5-1.5-Feynman-GRPO-vLLM-Merged"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", trust_remote_code=True)

generator = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    device="cuda"  # or "cpu"
)

prompt = "Explain quantum entanglement from first principles."
output = generator(prompt, max_new_tokens=200, temperature=0.7, top_p=0.9)
print(output[0]["generated_text"])