This model is a fine-tuned version of microsoft/deberta-v3-base on the Quora Question Pairs dataset. It uses a cross-encoder architecture to determine whether two questions are semantically equivalent.

Key Features:

Cross-encoder architecture for superior accuracy
Probability calibration for reliable confidence estimates
Robust handling of missing/empty questions
Production-ready inference pipeline

Performance

Metric	Value
ROC AUC	97.59%
Training Loss	0.116
Validation Loss	0.214

Intended Use

Primary Use Cases:

Question deduplication systems
Semantic similarity detection
Content moderation for duplicate questions
Search and retrieval systems

Out-of-Scope Use:

General text similarity (model is optimized for questions)
Languages other than English
Longer texts (trained on max 128 tokens)

Usage

Basic Inference

python

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

# Load model and tokenizer
model_name = "fatihburakkaragoz/quora-cross-encoder"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

# Example usage
question1 = "How do I learn Python programming?"
question2 = "What's the best way to learn Python?"

# Tokenize and predict
inputs = tokenizer(question1, question2, 
                  truncation=True, padding=True, 
                  max_length=128, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)
    logits = outputs.logits
    probability = torch.softmax(logits, dim=-1)[0, 1].item()

print(f"Duplicate probability: {probability:.3f}")

With Probability Calibration (Recommended)

For the most accurate confidence estimates, use the included calibrator:

python

import joblib
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

# Load model, tokenizer, and calibrator
model_name = "fatihburakkaragoz/quora-cross-encoder"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

# Note: Download the calibrator separately from the model repository
calibrator = joblib.load("deberta_cal.pkl")

def predict_duplicate(question1, question2):
    # Get raw prediction
    inputs = tokenizer(question1, question2, truncation=True, 
                      padding=True, max_length=128, return_tensors="pt")
    
    with torch.no_grad():
        logits = model(**inputs).logits
        raw_prob = torch.sigmoid(logits[0, 1]).item()
    
    # Apply calibration for better confidence estimates
    calibrated_prob = calibrator.predict_proba([[raw_prob]])[0, 1]
    return calibrated_prob

# Example
prob = predict_duplicate("How to cook pasta?", "What's the best pasta recipe?")
print(f"Calibrated duplicate probability: {prob:.3f}")

Training Details

Training Data

Dataset: Quora Question Pairs (~400K question pairs)
Split: 90% training, 10% validation (stratified)
Preprocessing: Missing values filled with empty strings

Training Configuration

Base Model: microsoft/deberta-v3-base
Architecture: Cross-encoder with sequence classification head
Max Length: 128 tokens
Batch Size: 8 per device (with gradient accumulation)
Learning Rate: 2e-5
Epochs: 3
Optimizer: AdamW
Precision: FP16

Training Results

Epoch	Training Loss	Validation Loss	ROC AUC
1	0.219	0.211	0.972
2	0.171	0.198	0.976
3	0.116	0.214	0.976

Technical Details

Model Architecture

Type: Cross-encoder (both questions processed together)
Advantage: Higher accuracy than bi-encoder approaches
Trade-off: Slower inference than bi-encoders

Probability Calibration

This model includes a calibration component that improves probability estimates:

Method: Logistic Regression on validation predictions
Benefit: More reliable confidence scores for production use
File: deberta_cal.pkl (included in repository)

Limitations and Bias

Limitations:

Optimized for English question pairs only
Performance may degrade on very long questions (>128 tokens)
Training data reflects Quora user demographics and question patterns

Bias Considerations:

Model inherits biases from DeBERTa base model and Quora dataset
May perform differently across question domains/topics
Evaluation primarily on question similarity, not general text

Citation

If you use this model, please cite:

bibtex

@misc{deberta-v3-quora-question-pairs,
  title={DeBERTa-v3 for Quora Question Pairs Duplicate Detection},
  author={Fatih Burak Karagöz},
  year={2025},
  url={https://huggingface.co/fatihburakkaragoz/quora-cross-encoder}
}

Acknowledgments

Microsoft Research for DeBERTa-v3-base
Quora for the Question Pairs dataset
Hugging Face for the transformers library