This model is a 4-bit quantized version of the original microsoft/DialoGPT-medium model, optimized for reduced memory usage while maintaining performance.

Quantization Details

Quantization Type: 4-bit
bnb_4bit_quant_type: nf4
bnb_4bit_use_double_quant: True
bnb_4bit_compute_dtype: bfloat16
bnb_4bit_quant_storage: uint8
Original Footprint: 1444.46 MB (FLOAT32)
Quantized Footprint: 384.75 MB
Memory Reduction: 73.4%

Usage

python

from transformers import AutoModelForSequenceClassification, AutoTokenizer, BitsAndBytesConfig

model_name = "DialoGPT-medium-bnb-4bit-nf4"
model = AutoModelForSequenceClassification.from_pretrained(
    "your-username/microsoft/DialoGPT-medium",  # Replace with your HF username
    quantization_config=BitsAndBytesConfig(load_in_4bit=True)
)
tokenizer = AutoTokenizer.from_pretrained("your-username/microsoft/DialoGPT-medium")

manu02/dialogpt-medium-bnb-4bit-nf4-dq

DialoGPT-medium (Quantized)

Description

Quantization Details

Usage

manu02/
dialogpt-medium-bnb-4bit-nf4-dq