You can deploy this model and start earning money today!

mariadelcarmenramirez/
Llama-3.1-8B-QLoRA-ArtTherapy

Name: Llama-3.1-8B-QLoRA-ArtTherapy
Availability: InStock

Views

No views yet

Llama-3.1-8B-QLoRA-ArtTherapy

A QLoRA fine-tuned version of meta-llama/Llama-3.1-8B-Instruct specialized for art therapy dialogue. The model was trained to support structured, phase-aware therapeutic conversations using art therapy techniques and methodologies.

Model Details

Model Description

This model is a parameter-efficient fine-tune of Llama-3.1-8B-Instruct using QLoRA (Quantized Low-Rank Adaptation). It was trained on a curated art therapy conversation dataset structured across four therapeutic phases, enabling the model to generate contextually appropriate, phase-sensitive responses consistent with art therapy practice.

The adapter was trained using 2× NVIDIA Tesla T4 GPUs with 4-bit quantization (NF4) and data-parallel training via Accelerate.

Developed by: María del Carmen Ramírez
Model type: Causal Language Model (fine-tuned with QLoRA / PEFT)
Language(s): Spanish
License: Llama 3.1 Community License (inherits from base model)
Base model: meta-llama/Llama-3.1-8B-Instruct
Training dataset: mariadelcarmenramirez/art-therapy-data

Uses

Direct Use

This model can be used out-of-the-box for art therapy-oriented conversational tasks. It is suited for generating therapist-style responses in dialogue systems that follow a structured therapeutic arc (intake, exploration, reflection, closure).

Downstream Use

The model can be integrated into:

Mental health support applications with art therapy modules
Educational tools for training art therapy students
Research platforms studying AI-assisted therapeutic dialogue
Chatbot frameworks requiring structured, phase-aware conversation management

Out-of-Scope Use

This model is not intended to replace licensed art therapists or mental health professionals. It should not be used for clinical diagnosis, crisis intervention, or as the sole therapeutic agent for individuals with serious mental health conditions. Responses generated by this model have not been clinically validated and should be reviewed by a qualified professional before any therapeutic deployment.

How to Get Started with the Model

python

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

base_model_id = "meta-llama/Llama-3.1-8B-Instruct"
adapter_id = "mariadelcarmenramirez/Llama-3.1-8B-QLoRA-ArtTherapy"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)

base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.float16,
    device_map="auto",
)

model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

messages = [
    {"role": "system", "content": "You are an art therapy assistant. <FASE_1>"},
    {"role": "user", "content": "I'd like to start with something creative today."}
]

input_ids = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

with torch.no_grad():
    output = model.generate(input_ids, max_new_tokens=256, temperature=0.7, do_sample=True)

print(tokenizer.decode(output[0][input_ids.shape[-1]:], skip_special_tokens=True))

Training Details

Training Data

The model was trained on mariadelcarmenramirez/art-therapy-data, a dataset of 9,572 structured art therapy dialogue examples. Each example contains a prompt field (system + user messages) and a completion field (assistant response), formatted using Llama 3's chat template.

The dataset is organized into four therapeutic phases tagged via <FASE_N> markers in the system prompt. Balanced sampling (700 per phase) was applied to prevent class imbalance. The resulting 2,800 examples were shuffled and split 90/10 into train (2,520) and eval (280) sets.

Training Procedure

Preprocessing

The chat template from meta-llama/Llama-3.1-8B-Instruct was applied to format each dialogue into a single text string. The maximum sequence length was dynamically set to the 95th percentile of training sequence lengths, yielding MAX_SEQ_LENGTH = 546 tokens.

Training Hyperparameters

Parameter	Value
Base model	Llama-3.1-8B-Instruct
Fine-tuning method	QLoRA (4-bit NF4)
Training epochs	3
Learning rate	5e-5
LR scheduler	Cosine
Warmup steps	23 (5% of total)
Per-device batch size	1
Gradient accumulation	8
Effective batch size	16 (2 GPUs × 1 × 8)
Weight decay	0.01
Optimizer	paged_adamw_8bit
Max gradient norm	0.3
LoRA rank (r)	32
LoRA alpha	64
LoRA dropout	0.1
LoRA target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Trainable parameters	83,886,080 (1.03% of total)
Total parameters	~8.1B
Quantization	4-bit NF4 (double quant, fp16 compute)
Training precision	fp16 (via BitsAndBytes)
Gradient checkpointing	Enabled
Sequence packing	Disabled

Speeds, Sizes, Times

Metric	Value
Training hardware	2× NVIDIA Tesla T4 (15.6 GB each)
Training platform	Kaggle (multi-GPU DDP via Accelerate)
Total training time	~3 hours 46 minutes
Train samples/second	0.556
Total FLOPs	151,508,871 GF
Adapter size (on disk)	~336 MB

Evaluation

Testing Data, Factors & Metrics

Factors

Evaluation was performed at regular checkpoints (every 79 steps, approximately twice per epoch) across all four therapeutic phases present in the held-out set. No disaggregated evaluation by phase was conducted separately.

Metrics

The primary evaluation metric used for model selection was evaluation loss (eval_loss). Additional logged metrics include:

eval_mean_token_accuracy: per-token prediction accuracy on the eval set
eval_entropy: average prediction entropy
eval_num_tokens: cumulative tokens evaluated

Results

Checkpoint (epoch)	eval_loss	eval_mean_token_accuracy
0.50	1.437	62.52%
1.00	1.358	64.08%
1.50	1.340	64.65%
2.00	1.313	65.30%
2.50	1.331	65.49%
3.00	1.332	65.42%

The best checkpoint was reached at epoch 2.0 (eval_loss = 1.313). Training completed all 3 epochs (474 steps total) with the epoch-2 checkpoint selected as the best model via load_best_model_at_end=True. Final training loss was 1.238.