This model is a fine-tuned version of Google's Gemma-3 270M, specifically trained for mental health conversational support using Cognitive Behavioral Therapy (CBT) principles. The model has been trained on 5M+ tokens of high-quality mental health conversational data to provide empathetic, supportive, and therapeutically-informed responses.

Developed by: Saurav Kumar Srivastava

Model Details

Base Model: google/gemma-3-270m
Model Size: 270M parameters
Training Data: 5M+ tokens of CBT-based therapeutic conversations
Training Method: LoRA fine-tuning using Unsloth
Quantization: BF16 GGUF format available
License: MIT

Training Configuration

The model was fine-tuned using the following specifications:

LoRA Rank (r): 8
LoRA Alpha: 8
Target Modules: All attention and MLP modules
Batch Size: 2 (per device) with 4 gradient accumulation steps
Learning Rate: 2e-4
Training Steps: 30 (optimized for efficiency)
Optimizer: AdamW 8-bit
Framework: Unsloth + TRL SFTTrainer

Intended Use

Primary Use Cases

Mental Health Support: Providing empathetic conversations and CBT-based guidance
Therapeutic Assistance: Supporting individuals with anxiety, depression, and stress management
Educational Tool: Teaching CBT techniques and mental health awareness
Research: Studying conversational AI in mental health applications

Limitations

Not a Replacement for Professional Help: This model should not replace licensed mental health professionals
Crisis Situations: Not suitable for handling severe mental health crises or suicidal ideation
General Limitations: As with all language models, may occasionally generate inappropriate or inaccurate responses

Usage

Basic Inference

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained("Skshackster/gemma3-270m-mental-health-fine-tuned-gguf")
tokenizer = AutoTokenizer.from_pretrained("Skshackster/gemma3-270m-mental-health-fine-tuned-gguf")

# Prepare conversation
messages = [{
    "role": "user",
    "content": [{"type": "text", "text": "I've been feeling really anxious lately about work."}]
}]

# Generate response
text = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=128,
        temperature=1.0,
        top_p=0.95,
        top_k=64,
        do_sample=True
    )

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Recommended Inference Settings

Temperature: 1.0
Top-p: 0.95
Top-k: 64
Max New Tokens: 64-256 (depending on desired response length)

Training Data

The model was trained on a carefully curated dataset of mental health conversations incorporating:

CBT-based therapeutic dialogues
Empathetic response patterns
Crisis de-escalation techniques
Mindfulness and coping strategies
Educational mental health content

Data Volume: 5M+ tokens of high-quality conversational data

Evaluation and Performance

The model demonstrates strong performance in:

Empathetic response generation
CBT technique application
Maintaining therapeutic conversation flow
Appropriate boundary setting
Educational content delivery

Ethical Considerations

Safety Measures

Trained to redirect users to professional help when appropriate
Designed to avoid giving specific medical advice
Incorporates safety guidelines for mental health conversations
Includes appropriate disclaimers about professional treatment

Bias and Fairness

Efforts made to ensure inclusive and culturally sensitive responses
Regular evaluation for potential biases in mental health recommendations
Continuous monitoring for harmful or inappropriate outputs

Technical Specifications

Architecture: Gemma-3 (Transformer-based)
Context Length: 4000 tokens
Precision: BF16
Hardware Requirements: Compatible with consumer GPUs (4GB+ VRAM recommended)
Inference Speed: Optimized for real-time conversation

Files and Formats

Standard Model: PyTorch format compatible with Transformers library
GGUF Format: Available for llama.cpp and Ollama integration
Quantization: BF16 precision maintained for quality

Citation

If you use this model in your research or applications, please cite:

bibtex

@misc{srivastava2025gemma3mentalhealth,
  title={Gemma-3 270M Mental Health Fine-tuned Model},
  author={Saurav Kumar Srivastava},
  year={2025},
  howpublished={\url{https://huggingface.co/Skshackster/gemma3-270m-mental-health-fine-tuned-gguf}},
}

Contact and Support

Developer: Saurav Kumar Srivastava

For questions, issues, or collaboration inquiries, please open an issue in the model repository

Acknowledgments

Google for the Gemma-3 base model
Unsloth for the efficient fine-tuning framework
Mental Health Community for supporting ethical AI development in therapeutic applications

Disclaimer

This model is designed for educational and supportive purposes only. It should not be used as a substitute for professional mental health treatment. If you are experiencing a mental health crisis, please contact a licensed mental health professional or emergency services immediately.