You can deploy this model and start earning money today!

ykshrestha/
personalmail-rl-qwen

Name: personalmail-rl-qwen
Availability: InStock

Views

No views yet

📧 PersonalMail-RL: Teaching LLMs to Handle Personal Emails Through Reinforcement Learning

Theme: #3.2 Personalized Tasks · OpenEnv Hackathon Apr 2026
Stack: OpenEnv · TRL GRPO · Unsloth · Qwen2.5-1.5B-Instruct
Demo: 🚀 Live on HuggingFace Spaces
Model: 🤗 ykshrestha/personalmail-rl-qwen
Code: 💻 GitHub Blog link: 💻 Blog Youtube Video: 💻Youtube Link PPT link: Link Training Scripts link: Link Blog link: Link

⚠️ To view live inference results in the demo, please select Nvidia A100 Large GPU in HuggingFace Space Settings before running.

🎯 The Problem

Personal emails are not equal. A message from your manager about a missed deadline is completely different from a dinner invite from a friend. Yet most LLMs treat every email the same — they reply generically, miss urgency signals, use the wrong tone, and sometimes reply to emails that should never be replied to (like spam).

What we built: A 2-step RL environment where a model learns to first understand an email, then respond to it appropriately — rewarded by 6 independent verifiable functions.

Why RL? You cannot write ideal replies for every scenario in advance. But you can verify whether a reply is good: Does it have a proper greeting? Does it address the key points? Is the tone right? RL with verifiable rewards is the perfect fit.

🏗️ Architecture & Flow

┌─────────────────────────────────────────────────────────────┐
│                     PersonalMail-RL                          │
│                                                              │
│   Email Scenario (25 scenarios × 3 difficulties)            │
│          │                                                   │
│          ▼                                                   │
│   ┌─────────────┐   Step 1 Action    ┌──────────────────┐  │
│   │   Agent     │ ─────────────────► │  Classification  │  │
│   │ (Qwen2.5    │                    │  Reward (2 fns)  │  │
│   │  1.5B-RL)   │ ◄───────────────── │  Score: 0-1      │  │
│   │             │   Reward + Obs      └──────────────────┘  │
│   │             │                                           │
│   │             │   Step 2 Action    ┌──────────────────┐  │
│   │             │ ─────────────────► │  Reply Quality   │  │
│   │             │                    │  Reward (5 fns)  │  │
│   │             │ ◄───────────────── │  Score: 0-1      │  │
│   └─────────────┘   Reward + Done    └──────────────────┘  │
│                                                              │
│   Episode Total = 0.30 × Step1 + 0.70 × Step2              │
│                                                              │
│   OpenEnv (FastAPI) → TRL GRPO → Unsloth QLoRA             │
└─────────────────────────────────────────────────────────────┘

Step 1 — Email Classification

The agent receives the email and outputs:

json

{
  "urgency": "high | medium | low",
  "category": "work | personal | social | spam",
  "requires_reply": true | false,
  "reason": "2-3 sentence explanation"
}

Step 2 — Reply Drafting

Using its own classification as context, the agent drafts:

json

{
  "tone": "professional | friendly | assertive | apologetic | none",
  "reply_text": "Full email reply here"
}

🌍 Environment Design

Built with OpenEnv + FastAPI — fully OpenEnv spec compatible.

OpenEnv Interface

python

env.reset(scenario_id=None)   # Start new episode, receive email
env.step(action)               # Submit classification or reply → get reward
env.state()                    # Current episode state for monitoring

Scenario Dataset

25 handcrafted real-world email scenarios × 3 difficulty levels:

Difficulty	Count	Examples
🟢 Easy	10	Clear work emails, obvious spam, simple social invites
🟡 Medium	8	Ambiguous tone, mixed urgency, passive-aggressive messages
🔴 Hard	7	Conflict emails, emotional situations, multi-intent messages

Curriculum Learning

Training starts with easy scenarios only → progressively adds medium → then hard. This ensures the model sees successful trajectories early and avoids zero-reward stalls.

🏆 7 Independent Reward Functions

Using multiple independent reward functions is the core defense against reward hacking. Here is every function implemented:

Step 1 Rewards — Classification (30% of Episode)

Reward 1: classification_accuracy — 21% of Episode Total

✅ Urgency correct (high/medium/low)            → +0.34
✅ Category correct (work/personal/social/spam) → +0.33
✅ Requires_reply correct (true/false)          → +0.33

Reward 2: classification_format — 9% of Episode Total

✅ All 4 required JSON fields present           → base score
✅ Reason field ≥ 5 words                       → +0.10 bonus

Step 2 Rewards — Reply Quality (70% of Episode)

Reward 3: reply_relevance — 28% of Episode Total

✅ Must-include keywords matched in reply
   Score = keywords_matched / total_keywords
✅ Spam emails auto-score 1.0 (correct to not reply)

Reward 4: reply_format — 24.5% of Episode Total

✅ Greeting present (Dear/Hi/Hello...)          → +0.30
✅ Body has ≥ 2 substantive sentences           → +0.35
✅ Closing present (Best/Regards/Thanks...)     → +0.35

Reward 5: reply_length — 17.5% of Episode Total

Target: 40–150 words
✅ 40-150 words  → 1.0  (perfect)
⚠️ 25-40 words  → 0.7  (short)
⚠️ 150-250 words → 0.7  (long)
❌ <25 words    → 0.3  (too short)
❌ >250 words   → 0.4  (too long)

Reward 6: tone_matching — 15% of Episode Total

Tone detected via lexical signals:
professional → "please", "kindly", "regards", "sincerely"
friendly     → "hey", "cheers", "sounds great", "would love"
assertive    → "I expect", "immediately", "by end of"
apologetic   → "I apologize", "deeply sorry", "I take full responsibility"

Reward 7: reply_clarity — 15% of Episode Total

✅ Actionable signals (timelines, action verbs, coordination phrases)
❌ Penalizes vague filler ("noted", "will revert", "as soon as possible")
✅ Unique word ratio ≥ 0.55 (not repetitive)

Episode Total Formula

Episode = 0.30 × Step1_total + 0.70 × Step2_total

Reply quality is weighted 70% because drafting a contextually appropriate reply is the harder, more valuable task.

🛡️ Anti-Reward-Hacking Measures

1. 7 Independent Signals — Cannot game any one without the others catching it.

2. Timeout Penalty — Episodes exceeding 120 seconds receive −0.5 penalty.

3. Duplicate Action Detection — Repeated identical actions are penalized.

4. Spam Safeguard — Spam emails reward not replying. Replying to spam scores 0.

5. Reply Clarity Guard — Penalizes vague "polite-only" hacks like "noted, will revert."

6. Real Discovery During Training:

During GRPO training, we discovered the model learned to output "reason": ["some text"] (a list) instead of "reason": "some text" (a string) to exploit our format reward. The field was "present" but not actually a valid string. We identified this as reward hacking and fixed reward functions to be type-safe — a real RL failure mode caught and resolved.

🔬 Training Pipeline

Stack

Qwen2.5-1.5B-Instruct (base)
        ↓
  Unsloth 4-bit QLoRA (memory efficiency)
        ↓
  TRL GRPOTrainer
        ↓
  PersonalMail-RL Environment (live reward scoring)
        ↓
  LoRA Adapter → ykshrestha/personalmail-rl-qwen

Training Configuration

yaml

model: Qwen2.5-1.5B-Instruct
lora_rank: 16
quantization: 4-bit QLoRA
rollouts: 6 per step
steps: 200
lr: 5e-5
save: adapter (no naive merge)

Training Script

bash

# Run in Google Colab
PYTHONPATH=./personalmail-rl-opt \
  python training/train_grpo.py

📊 Evaluation Results — Before vs After Training

Evaluated on 12 scenarios comparing Baseline Qwen2.5-1.5B vs our GRPO-trained adapter.
Model trained

Colab Evaluation Output

json

{
  "num_scenarios": 12,
  "baseline": {
    "classification": 0.4738,
    "reply_total": 0.3874,
    "reply_format": 0.4542,
    "reply_relevance": 0.3611,
    "reply_length": 0.4500,
    "tone_matching": 0.1949,
    "episode_total": 0.4133
  },
  "trained": {
    "classification": 0.2905,
    "reply_total": 0.7664,
    "reply_format": 0.8917,
    "reply_relevance": 0.7223,
    "reply_length": 0.8333,
    "tone_matching": 0.6207,
    "episode_total": 0.6237
  },
  "improvement": {
    "reply_format": "+0.4375 ✅",
    "tone_matching": "+0.4258 ✅",
    "reply_relevance": "+0.3612 ✅",
    "reply_length": "+0.3833 ✅",
    "reply_total": "+0.3790 ✅",
    "episode_total": "+0.2104 ✅"
  }
}

Results Summary Table

Metric	Baseline	Trained	Improvement
Reply Format	0.454	0.892	+0.438 ✅
Tone Matching	0.195	0.621	+0.426 ✅
Reply Length	0.450	0.833	+0.383 ✅
Reply Relevance	0.361	0.722	+0.361 ✅
Reply Total	0.387	0.766	+0.379 ✅
Episode Total	0.413	0.624	+0.210 ✅
Classification	0.474	0.291	-0.183 ⚠️

What Improved and Why

Reply quality improved dramatically across all metrics:

+44% reply format — Model learned to always include greeting + body + closing
+43% tone matching — Model learned to match professional/friendly/apologetic tone correctly
+38% reply length — Model learned to write replies of appropriate length
+36% reply relevance — Model learned to address actual content of the email
+21% overall episode — Significant improvement in end-to-end email handling

Classification slightly degraded (expected tradeoff): The base Qwen model already classifies emails reasonably. GRPO focused reward signal on reply quality (70% weight) so the model optimized for the harder, more valuable task. This is an expected behavior in weighted RL.

Evaluation Command

bash

PYTHONPATH=./personalmail-rl-opt \
  python training/evaluate_before_after.py \
  --baseline_model "Qwen/Qwen2.5-1.5B-Instruct" \
  --trained_model "ykshrestha/personalmail-rl-qwen" \
  --num_scenarios 12 \
  --out_json eval_results.json

🖥️ Live Demo Screenshots

Live Demo — Step 1: Email Classification

The agent classifies the email and receives immediate reward feedback. Step 1 Reward: 0.762 on a conflict resolution email.

Live Demo — Step 2: Reply Drafting

The agent drafts a contextually appropriate reply. Episode Complete with Total Reward: 0.681

Before vs After Comparison

Side-by-side comparison of Baseline vs GRPO-Trained model on the same email.

Megha: Conflict email — Meeting overlaps with client call

	Baseline	Trained (GRPO)
Reply	"Hi Megha, I understand there's an overlap... Could we possibly rearrange things..."	"Hello Megha, I understand your concern about overlapping commitments. Let's try to find a mutually convenient time... If it's possible, we could adjust the design sync to another day or hour that doesn't clash with the client call. Please let me know if there's anything else I can assist with regarding scheduling adjustments. Best regards, [Your Name]"
Episode Reward	0.614	0.694
Improvement		+0.080 ✅

The trained model's reply is more structured, uses proper closing ("Best regards"), is more specific about solutions, and scores higher on clarity and format.

Environment Info Tab

Full checklist of all hackathon requirements — all green ✅

✅ Hackathon Requirements Checklist

Requirement	Status
OpenEnv-compatible FastAPI server	✅
TRL GRPO training script	✅
Unsloth QLoRA efficiency	✅
Google Colab notebook	✅
7 independent reward functions	✅
Clarity guardrail against vague "polite-only" hacks	✅
Anti-reward-hacking measures	✅
Process supervision (2-step)	✅
Interactive demo UI	✅
Before/After comparison with real rewards	✅
Docker deployment ready	✅
HuggingFace Spaces deployed	✅
Mini blog post	✅

🚀 Deployment

HuggingFace Space

URL: https://huggingface.co/spaces/ykshrestha/personalmail-rl-demo
Hardware: Nvidia A100 GPU (required for inference)
Backend: FastAPI (OpenEnv compatible)
Frontend: Interactive HTML UI

⚠️ Important: Select Nvidia A100 Large in Space Settings to enable live model inference. CPU mode will not support float16 inference.

Run Locally

bash

git clone https://github.com/YASHASWINIKSHRESTHA/personalmail-rl-opt
cd personalmail-rl-opt/personalmail-rl-opt
pip install -r requirements.txt
uvicorn server:app --host 0.0.0.0 --port 7863

Docker

bash

docker build -t personalmail-rl .
docker run -p 7863:7863 personalmail-rl

📁 Repository Structure

personalmail-rl-opt/
├── env/
│   ├── environment.py           # OpenEnv reset/step/state
│   ├── rewards.py               # 7 independent reward functions
│   ├── scenarios.py             # 25 email scenarios × 3 difficulties
│   └── models.py                # Pydantic schemas
├── training/
│   ├── train_grpo.py            # GRPO training script
│   ├── evaluate_before_after.py # Before/after evaluation
│   └── PersonalMailRL_Colab.ipynb  # Full training notebook
├── server.py                    # FastAPI OpenEnv server
├── ui/index.html                # Interactive demo UI
├── Dockerfile                   # HuggingFace Space deployment
└── openenv.yaml                 # OpenEnv configuration

🔗 Links

Resource	Link
🤗 Trained Model	ykshrestha/personalmail-rl-qwen
🚀 Live Demo Space	ykshrestha/personalmail-rl-demo
💻 GitHub Code	YASHASWINIKSHRESTHA/personalmail-rl-opt
📊 Eval Results	eval_v5.json — +0.21 episode total improvement