Five-class response sufficiency classifier using DeBERTa-v3 as a cross-encoder. Takes (objective, response) pairs as separate inputs with direct cross-attention between the two texts.

What changed in v2

v1 (KingTechnician/osmosis-crossencoder-joint) was trained with the Yahoo response side taken from the raw answer column. Many Yahoo answers begin with a near-verbatim echo of the question, which gave the cross-encoder a trivial lexical-overlap shortcut to exploit. v1's reported ~82.5% Yahoo accuracy reflected that shortcut, not the true task.

v2 parses (objective, response) out of the prepared text field on the dataset, which has echoes stripped. Triage used (objective, student_message) in both versions and was never affected.

Performance

Evaluation	Accuracy	Macro F1
Yahoo within-domain	57.5%	41.7%
Triage held-out	99.7%	99.7%

Per-class (Yahoo held-out, 7,074 samples)

Class	Precision	Recall	F1	Support
ADDR_DIRECT	58.0%	65.3%	61.5%	2508
ADDR_PARTIAL	48.5%	43.4%	45.8%	1052
NOADDR_ON	61.6%	65.5%	63.5%	2816
NOADDR_TANGENTIAL	53.0%	22.8%	31.9%	536
NOADDR_OFF	8.5%	4.3%	5.7%	162

Per-class (Triage held-out, 374 samples)

Class	F1
ADDR_DIRECT	99.3%
ADDR_PARTIAL	99.3%
NOADDR_ON	100%
NOADDR_TANGENTIAL	100%
NOADDR_OFF	100%

1 misclassification out of 374 Triage test samples.

Known limitations

NOADDR_OFF is effectively broken on Yahoo. Recall is 4.3%; of 162 true NOADDR_OFF samples, 95 are predicted as NOADDR_ON. The model almost never predicts NOADDR_OFF on open-domain text.
NOADDR_TANGENTIAL recall is 22.8% on Yahoo. Similar collapse pattern.
Triage is a synthetic, well-separated dataset. The 99.7% number is representative of structured classroom triage input, not open-domain text. Yahoo is the more realistic estimate for open-domain deployment.
Routing implication: Do not route based on predicted class for NOADDR_OFF / NOADDR_TANGENTIAL — those predictions are too rare to be useful as a routing signal. Uncertainty-based routing is required.

Usage

python

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_id = "KingTechnician/osmosis-crossencoder-joint-v2"
model = AutoModelForSequenceClassification.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)

labels = ["ADDR_DIRECT", "ADDR_PARTIAL", "NOADDR_ON", "NOADDR_TANGENTIAL", "NOADDR_OFF"]

objective = "What causes rain?"
response = "Rain forms when water vapor in the atmosphere condenses into droplets."

inputs = tokenizer(objective, response, return_tensors="pt", truncation=True, max_length=512)
with torch.no_grad():
    logits = model(**inputs).logits
    prediction = logits.argmax(dim=-1).item()

print(f"Prediction: {labels[prediction]}")

Architecture

Cross-encoder. The model processes [CLS] objective [SEP] response [SEP] as a single input, allowing full self-attention between objective and response tokens. This is essential for response sufficiency classification because the judgment depends on token-level alignment between what was asked and what was answered.

Training

Base model: MoritzLaurer/deberta-v3-base-zeroshot-v2.0 (NLI-pretrained)
Data: Yahoo (echo-stripped) + Triage, joint training
Epochs: 10, LR: 2e-5 with linear warmup (10%), batch=16, max_len=512
Loss: Cross-entropy with class weights [0.57, 1.31, 0.52, 2.41, 6.84]

Labels

Label	Description
ADDR_DIRECT	Response directly and completely addresses the objective
ADDR_PARTIAL	Response partially addresses the objective
NOADDR_ON	Response is on-topic but does not address the objective
NOADDR_TANGENTIAL	Response is tangentially related to the objective
NOADDR_OFF	Response is completely off-topic