This binary classifier automates the first step of the Motivational Interviewing Treatment Integrity (MITI) 4.2 coding process: determining whether a therapist utterance should be coded or excluded from analysis.
What is MITI 4.2?
The Motivational Interviewing Treatment Integrity (MITI) 4.2 is a behavioral coding system used to assess fidelity to Motivational Interviewing (MI). In MITI coding:
First step: Coders review each therapist utterance and decide if it should be coded or not coded
Coded utterances: Substantive therapist statements that warrant behavioral coding
Second step: Coded utterances are then classified into behavior codes (Giving Information, Persuade, Question, Simple Reflection, Complex Reflection, etc.)
This model automates the first step, providing a foundation for full MITI 4.2 automation.
Model Architecture
Base Model: Qwen/Qwen3-Embedding-0.6B
Task Type: Binary sequence classification
Input: Therapy session context + last therapist utterance
Output: Binary prediction (coded=1, not_coded=0)
Max Sequence Length: 3000 tokens
Training Framework: HuggingFace Transformers with Flash Attention 2
Precision: bfloat16 for efficient inference
Training Data
The model was trained on multilabel classifier dataset with annotations from two expert coders:
Annotator Profiles
AJ (Expert Annotator): MI and MITI 4.2 trained expert with extensive coding experience
SJ (Beginner Annotator): Psychologist who has reached beginner-level proficiency with ICC (Intra-Class Correlation) > 0.86 with the expert annotator AJ
Data Splits
Training Set: 80% of data (stratified by label)
Validation Set: 10% of data (stratified by label)
Test Set: 10% of data (stratified by label)
Total Test Examples: 3,776 utterances
Class Distribution
The dataset exhibits class imbalance (more coded than not coded utterances), which is addressed through:
Balanced class weighting in the loss function
Stratified train/val/test splits
Training Configuration
python
1- Model: Qwen/Qwen3-Embedding-0.6B
2- Batch Size:123- Learning Rate:6e-54- Epochs:20(with early stopping)5- Warmup Ratio:0.16- Weight Decay:0.017- LR Scheduler: Cosine
8- Optimization: Weighted Cross-Entropy Loss
9- Early Stopping: Patience=3, Threshold=0.00110- Metric for Model Selection: F1 Macro
Performance Metrics
Test Set Results
Metric
Score
Overall Accuracy
97.99%
F1 Score (Macro)
94.79%
Precision (Macro)
96.51%
Recall (Macro)
93.23%
Per-Class Performance
Coded Class (Positive Class, Label=1)
F1 Score: 98.87%
Precision: 98.37%
Recall: 99.37%
Not Coded Class (Negative Class, Label=0)
F1 Score: 90.71%
Precision: 94.64%
Recall: 87.09%
Confusion Matrix (Test Set)
Predicted
Not_Coded Coded
Actual
Not_Coded 371 55
Coded 21 3329
Interpretation:
True Negatives (Not Coded → Not Coded): 371 (87.09%)
False Positives (Not Coded → Coded): 55 (12.91%)
False Negatives (Coded → Not Coded): 21 (0.63%)
True Positives (Coded → Coded): 3329 (99.37%)
The model shows excellent performance in identifying coded utterances (99.37% recall) with slightly lower recall for not coded utterances (87.09%). This bias toward coding is conservative and appropriate for clinical applications, as it's preferable to over-code than miss substantive therapeutic statements.
Calibration and Optimal Thresholds
The model has been calibrated to find optimal decision thresholds for different use cases. While the default threshold of 0.5 works well, alternative thresholds can optimize specific performance metrics:
Threshold Recommendations
Based on validation set analysis (see calibration_results.json for full details):
Use Case
Threshold
F1 Macro
Recommendation
Default (0.5)
0.5000
94.65%
Currently used by the model
Max F1
0.2773
94.68%
Optimizes overall F1 score - marginal improvement over default
Balanced P/R
0.9805
94.33%
Equalizes precision and recall
Use Case Specific Recommendations
Training & Education: Use high recall threshold (lower threshold values) to catch all potentially codeable utterances, ensuring learners don't miss substantive statements
Research: Use Max F1 threshold (0.2773) or default (0.5) for optimal overall performance and balanced metrics
Quality Assurance: Use high precision threshold (higher threshold values) to minimize false positives and reduce manual review burden
How to Use Custom Thresholds
The model outputs probabilities for both classes. To use a custom threshold:
python
1# Get probabilities2outputs = model(**inputs)3probabilities = torch.softmax(outputs.logits, dim=1)[0]4prob_coded = probabilities[1].item()56# Apply custom threshold7threshold =0.2773# Max F1 threshold8prediction ="coded"if prob_coded >= threshold else"not_coded"
Note: The calibration analysis includes ROC curves, Precision-Recall curves, and per-annotator threshold analysis. See calibration_results.json for comprehensive metrics at various threshold values.
Using Annotator Information for Inference
During inference, you can specify which annotator style to emulate by including annotator information in the input:
Annotator Emulation
python
1# Emulate expert annotator (AJ)2text ="""Task: Decide if the last therapist utterance should be coded or not.
3Annotated by: AJ
4[Your context and utterance here]"""56# Emulate beginner annotator (SJ)7text ="""Task: Decide if the last therapist utterance should be coded or not.
8Annotated by: SJ
9[Your context and utterance here]"""
Why this matters:
AJ emulation: Provides expert-level coding decisions aligned with extensive MI/MITI training
SJ emulation: Provides beginner-level proficiency coding with ICC > 0.86 correlation with expert
This allows users to choose the level of coding rigor appropriate for their use case (e.g., research vs. training)
Usage
Basic Inference
python
1from transformers import AutoTokenizer, AutoModelForSequenceClassification
2import torch
34# Load model and tokenizer5model_name ="Lekhansh/qwen_nc_classifier"6tokenizer = AutoTokenizer.from_pretrained(model_name)7model = AutoModelForSequenceClassification.from_pretrained(8 model_name,9 torch_dtype=torch.bfloat16,10 attn_implementation="flash_attention_2"11)12model.eval()1314# Prepare input15context ="""Patient: I've been trying to quit smoking but it's really hard.
16Therapist: Tell me more about what makes it difficult.
17Patient: Well, I smoke when I'm stressed at work.
18Therapist: """1920utterance ="Mm-hmm."2122# Format with annotator info (emulate expert)23text =f"""Task: Decide if the last therapist utterance should be coded or not.
24Annotated by: AJ
25Context:
26{context}2728Last Therapist Utterance: {utterance}"""2930# Tokenize and predict31inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=3000)32inputs ={k: v.to(model.device)for k, v in inputs.items()}3334with torch.no_grad():35 outputs = model(**inputs)36 logits = outputs.logits
37 predicted_class = torch.argmax(logits, dim=1).item()38 probabilities = torch.softmax(logits, dim=1)[0]3940# Interpret results41label_map ={0:"not_coded",1:"coded"}42prediction = label_map[predicted_class]43confidence = probabilities[predicted_class].item()4445print(f"Prediction: {prediction}")46print(f"Confidence: {confidence:.2%}")47print(f"Not Coded Probability: {probabilities[0]:.2%}")48print(f"Coded Probability: {probabilities[1]:.2%}")
Batch Inference
See demo_inference.py for a complete batch inference example with multiple utterances.
Limitations and Considerations
Training Data Scope: Model trained on specific therapy session formats; performance may vary with different conversational structures
Context Dependency: Model relies on conversational context; single utterances without context may yield less reliable predictions
Class Imbalance Effects: Higher recall for coded class (99.37%) vs not coded class (87.09%) reflects training data distribution
Annotator Variance: While ICC > 0.86 indicates good agreement, some coding decisions remain subjective
Domain Specificity: Optimized for Motivational Interviewing; may not generalize to other therapeutic modalities
Clinical Applications
This model can be used for:
Training and Education: Providing immediate feedback to MI learners on utterance coding
Quality Assurance: Automated pre-screening of therapy sessions before manual MITI coding
Research: Large-scale analysis of MI fidelity across multiple sessions and practitioners
Supervision: Assisting supervisors in reviewing trainee sessions efficiently
Citation
If you use this model in your research, please cite:
