This repository contains a BERT-base intent classifier fine-tuned on the BANKING77 dataset using a teacher pseudo-labeling distillation approach (Path 1). A fine-tuned BERT-large teacher generates predicted intent labels for each training example; the BERT-base student is then fine-tuned on these teacher-generated labels and evaluated on the original test set.

Model Details

Model Description

Developed by: <ANANT SRIVASTAVA / anant1213 >
Shared by: <ANANT SRIVASTAVA / anant1213 >
Model type: BertForSequenceClassification (BERT encoder + classification head)
Language(s): English
License: Apache-2.0 (inherits from base BERT license) :contentReference[oaicite:0]{index=0}
Finetuned from model: google-bert/bert-base-uncased :contentReference[oaicite:1]{index=1}
Teacher model (used for pseudo-labeling): google-bert/bert-large-uncased fine-tuned on BANKING77 :contentReference[oaicite:2]{index=2}

Model Sources

Base student model: google-bert/bert-base-uncased :contentReference[oaicite:3]{index=3}
Base teacher model: google-bert/bert-large-uncased :contentReference[oaicite:4]{index=4}
Dataset: PolyAI/banking77 :contentReference[oaicite:5]{index=5}

Uses

Direct Use

Classifying short banking-domain queries into one of 77 intent labels (e.g., “card payment reversed”, “balance not updated”, etc.). :contentReference[oaicite:6]{index=6}

Downstream Use

As a lightweight intent router for customer support workflows
As a feature extractor (CLS embedding) for downstream classifiers (requires custom code)
As a student initialization for further fine-tuning on a proprietary intent dataset (recommended)

Out-of-Scope Use

General-domain intent classification outside banking
Long-document classification (model trained on short queries)
High-stakes decision making without human verification

Bias, Risks, and Limitations

Domain limitation: Trained on banking queries; performance may degrade on other domains. :contentReference[oaicite:7]{index=7}
Teacher bias transfer: Student inherits systematic teacher errors since training labels are teacher predictions (pseudo labels).
Label noise: If the teacher is imperfect, pseudo labels introduce noise that can cap student performance.
No calibrated confidence guarantee: Softmax confidence may not reflect true correctness; calibrate if used in production.

Recommendations

If deploying in production, validate on in-domain data and monitor intent confusion pairs.
Consider adding a fallback intent (e.g., “unknown”) or thresholding for low-confidence predictions.

How to Get Started with the Model

Inference Example

python

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

repo_id = "anant1213/bert-base-banking77-student-pseudo-v1"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForSequenceClassification.from_pretrained(repo_id)

text = "I was charged twice for the same card payment"
inputs = tokenizer(text, return_tensors="pt", truncation=True)

with torch.no_grad():
    logits = model(**inputs).logits
pred_id = int(torch.argmax(logits, dim=-1))
print("Predicted label id:", pred_id)

Anant1213/bert-base-banking77-student-pseudo-v1

Model Card for anant1213/bert-base-banking77-student-pseudo-v1 (BERT Student — Pseudo-Label Distilled)

Summary