This model is a 4-bit quantized version of the original sentence-transformers/all-MiniLM-L6-v2 model, optimized for reduced memory usage while maintaining performance.

Quantization Details

Quantization Type: 4-bit
bnb_4bit_quant_type: nf4
bnb_4bit_use_double_quant: True
bnb_4bit_compute_dtype: bfloat16
bnb_4bit_quant_storage: uint8
Original Footprint: 90.99 MB (FLOAT32)
Quantized Footprint: 53.83 MB (UINT8)
Memory Reduction: 40.8%

Usage

python

from transformers import AutoModel, AutoTokenizer

model_name = "all-MiniLM-L6-v2-bnb-4bit-nf4"
model = AutoModel.from_pretrained(
    "manu02/all-MiniLM-L6-v2-bnb-4bit-nf4",
)
tokenizer = AutoTokenizer.from_pretrained("manu02/all-MiniLM-L6-v2-bnb-4bit-nf4", use_fast=True)

manu02/all-MiniLM-L6-v2-bnb-4bit-nf4-dq

all-MiniLM-L6-v2 (Quantized)

Description

Quantization Details

Usage

manu02/
all-MiniLM-L6-v2-bnb-4bit-nf4-dq