A LoRA adapter fine-tuned on empathetic dialogue data for emotionally supportive conversations with a warm, witty tone aimed at stressed tech workers.

Base Model

mistralai/Mistral-7B-Instruct-v0.3

Training Data

Fine-tuned on Estwld/empathetic_dialogues_llm, a dataset of multi-turn conversations covering 32 emotion categories, augmented with synthetic witty-empathetic examples for tone calibration. Each conversation includes a system prompt with the detected user emotion.

Training Details

Method: LoRA (Low-Rank Adaptation) via peft
Quantization: 4-bit NF4 via bitsandbytes
LoRA rank: 16, alpha: 32
Target modules: q_proj, v_proj, k_proj, o_proj
Epochs: 1
Learning rate: 2e-4 (cosine schedule)
Effective batch size: 16 (batch 4 x gradient accumulation 4)
Optimizer: paged AdamW 8-bit

Usage

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")
model = PeftModel.from_pretrained(base, "hyan/destress-empathy-lora")
tokenizer = AutoTokenizer.from_pretrained("hyan/destress-empathy-lora")

messages = [
    {"role": "system", "content": "You are a warm, witty companion for stressed tech workers.\nThe user is currently feeling: anxious."},
    {"role": "user", "content": "I have a big presentation tomorrow and I can't stop worrying about it."},
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Built for the Mistral Hackathon

Part of the DeStress project -- an empathetic AI voice companion for people experiencing stress and emotional challenges.