You can deploy this model and start earning money today!

saketgarodia1/
bert-it-ticket-student

Name: bert-it-ticket-student
Availability: InStock

text-embeddings-inference

endpoints_compatible

Views

No views yet

📘 DistilBERT IT Ticket Classifier (Student Model – Knowledge Distilled)

This is a DistilBERT student model fine-tuned via knowledge distillation from the teacher model:

👉 saketgarodia1/bert-IT-ticket-classifier-full

The goal is to create a lighter, faster IT ticket classifier while retaining high performance.

🔥 Results (Teacher vs Student)

⚠️ Replace the X values below once you finalize the test metrics.

Model	Accuracy	Macro F1	Notes
Teacher (BERT Base)	0.9374	0.9362	~110M params
Student (DistilBERT)	0.93X	0.93X	~67M params, ~40% smaller & faster

🧠 Distillation Setup

The student is trained using a combined loss:

[ \mathcal{L} = \alpha \cdot \text{CE}(\text{student}, \text{labels}) ;+; (1 - \alpha) \cdot \text{KL}(\text{student}, \text{teacher}) ]

Key settings:

α (alpha): 0.3 → more weight on hard labels
Temperature (T): 2 → KL on softened logits
Loss components:
- Cross-entropy (CE) between student logits and ground-truth labels
- KL divergence between softened student logits and teacher logits
Warmup ratio: 10%
Optimizer: AdamW
- Learning rate: 2e-5
- Weight decay: 0.01
Epochs: 3
Batch size: 16 / 32
Teacher: frozen; used only to generate logits (no gradient updates)

🧰 Intended Use

Classifying IT support tickets into 8 categories
Suitable for smaller GPUs / CPUs
Real-time or near real-time inference scenarios
Internal IT help desk routing, analytics, or triage dashboards

🚫 Limitations

Inherits potential biases from both BERT and DistilBERT
Assumes English-only input text
Not designed for long documents (> 512 tokens)
Domain-specific: trained on IT ticket text, may not generalize perfectly to other domains

📦 How to Use the Model

python

from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = "saketgarodia1/bert-it-ticket-student"

model = AutoModelForSequenceClassification.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)

text = "VPN not connecting to corporate WiFi"
inputs = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    logits = model(**inputs).logits

pred_class_id = logits.argmax(dim=-1).item()
print("Predicted class id:", pred_class_id)