A multi-label text classifier that assigns 15 semantic labels to group chat messages. Built for enriching chat data with structured content signals — useful for community analytics, newsletter curation, discussion ranking, and moderation tooling.

What It Does

Given a chat message, the model outputs 15 independent sigmoid scores (0.0 to 1.0), each indicating the probability that the message belongs to that category. Multiple labels can be active simultaneously.

Example:

"Has anyone tried fine-tuning DeBERTa for multi-label? I got NaN gradients in fp16."

Label	Score	Active
professional	0.94	Yes
question	0.91	Yes
experience_sharing	0.72	Yes
substantive	0.88	Yes
how_to	0.15	No
...	...	...

Labels

#	Label	Definition	Test F1	AUC-ROC
1	`professional`	Domain-relevant professional substance	0.839	0.882
2	`question`	Asks for information, help, or advice	0.889	0.977
3	`experience_sharing`	First-hand account of trying or building something	0.631	0.844
4	`resource`	Shares a link, tool, paper, or tutorial	0.746	0.928
5	`opinion`	Subjective take, prediction, or stance	0.626	0.853
6	`how_to`	Concrete tip, solution, or workaround	0.530	0.844
7	`humor`	Joke, meme, sarcasm, playful remark	0.347	0.792
8	`announcement`	Community event, meetup, release, group news	0.723	0.969
9	`off_group_topic`	Content unrelated to community's purpose	0.258	0.753
10	`reaction`	Acknowledgment, agreement, thanks, emoji-only	0.725	0.950
11	`substantive`	High information density, summary-worthy	0.787	0.963
12	`discussion_init`	Initiates a new topic or conversation	0.706	0.870
13	`emotional`	Highly emotional tone (excitement, frustration)	0.437	0.852
14	`disagreement`	Adversarial or confrontational disagreement	0.123	0.889
15	`positive_reinforcement`	Encouragement or gratitude	0.603	0.873

Quick Start

python

import json
import torch
import numpy as np
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_name = "eladlaor/chat-message-tagger-deberta-v3"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
model.eval()

# Load per-label thresholds (optimized on validation set)
from huggingface_hub import hf_hub_download
thresholds_path = hf_hub_download(model_name, "thresholds.json")
with open(thresholds_path) as f:
    thresholds = json.load(f)

# Label names in model output order
label_names = [
    "professional", "question", "experience_sharing", "resource", "opinion",
    "how_to", "humor", "announcement", "off_group_topic", "reaction",
    "substantive", "discussion_init", "emotional", "disagreement",
    "positive_reinforcement",
]

# Predict
text = "Has anyone tried fine-tuning DeBERTa? I got NaN gradients in fp16."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=256)

with torch.no_grad():
    logits = model(**inputs).logits[0]
    scores = torch.sigmoid(logits).numpy()

# Apply per-label thresholds
for name, score in zip(label_names, scores):
    threshold = thresholds.get(name, 0.5)
    active = "<<" if score > threshold else ""
    print(f"  {name:30s}  {score:.3f}  {active}")

Training

Base model: microsoft/deberta-v3-base (184M params)
Method: 2-phase fine-tuning: (1) head-only with frozen backbone (LR=1e-3), then (2) LoRA adaptation (rank=16, LR=2e-5)
Loss: BCEWithLogitsLoss with per-label pos_weight for rare labels
Precision: fp32 (DeBERTa v3 produces NaN in fp16/bf16)
Data: 3,842 training samples from professional tech community group chats (Hebrew translated to English)
Dataset: eladlaor/chat-message-multilabel

Evaluation

Metric	Value
Mean F1 (macro)	0.598
Hamming Loss	0.123
Subset Accuracy	0.193
Mean AUC-ROC	0.883
Mean ECE	0.136

Included Files

File	Description
`model.safetensors`	Merged model weights (LoRA baked in)
`config.json`	Model config with `problem_type=multi_label_classification`
`tokenizer.json`	Tokenizer with 7 special tokens ([URL], [MENTION], etc.)
`thresholds.json`	Per-label classification thresholds (optimized on validation set)
`label_taxonomy.json`	Label definitions and examples for downstream consumers
`evaluation.json`	Full per-label metrics on test set

Intended Use

Community analytics: Tag messages in Slack, Discord, Telegram, WhatsApp for content analysis
Newsletter curation: Identify high-value discussions for automated summaries
Moderation support: Flag off-topic content or heated disagreements
Discussion ranking: Enrich messages with semantic signals before LLM-based ranking

Limitations

English only. Trained on Hebrew messages translated to English. Non-English input will produce unreliable scores.
Tech community bias. Trained on professional tech community chats. Labels like "professional" and "how_to" are calibrated for tech discussions. May need fine-tuning for non-tech domains.
Weak labels: disagreement (F1=0.12), off_group_topic (F1=0.26), and humor (F1=0.35) have low F1 due to limited training data. Their AUC-ROC is strong (>0.75), so they rank correctly but thresholding is unreliable.
Not a content moderation tool. This model classifies content type, not toxicity or safety. Do not use it for content moderation decisions.

License

Apache 2.0

Citation

bibtex

@misc{chat-message-tagger-2026,
  author = {Elad Laor},
  title = {Chat Message Tagger: Multi-Label Classification for Group Chat Messages},
  year = {2026},
  publisher = {HuggingFace},
  url = {https://huggingface.co/eladlaor/chat-message-tagger-deberta-v3}
}