You can deploy this model and start earning money today!

dinhhungitsoft/
Qwen3-Embedding-4B-bnb4

Name: Qwen3-Embedding-4B-bnb4
Rating: 2.5 (33 reviews)

Views

No views yet

Qwen3-Embedding-4B-bnb4

This is a 4-bit quantized version of Qwen/Qwen3-Embedding-4B using BitsAndBytes NF4 quantization.

Model Details

Base Model: Qwen/Qwen3-Embedding-4B
Quantization Method: BitsAndBytes 4-bit (NF4)
Quantization Configuration:
- bnb_4bit_quant_type: "nf4"
- bnb_4bit_use_double_quant: True
- bnb_4bit_compute_dtype: float16
Model Size: ~2.5GB (down from ~8GB original)
Use Case: Text embedding for semantic search, retrieval, and similarity tasks

Performance

Tested on Vietnamese corpus (100 documents) with 20 queries:

Accuracy@1: 90%
MRR: 94.17%
Recall@3: 100%
Recall@5: 100%

Usage

With Transformers + BitsAndBytes

python

from transformers import AutoTokenizer, AutoModel
import torch

model_name = "dinhhungitsoft/Qwen3-Embedding-4B-bnb4"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(
    model_name,
    trust_remote_code=True,
    device_map="auto"
)

def embed(texts):
    inputs = tokenizer(texts, padding=True, truncation=True, return_tensors="pt", max_length=512)
    inputs = {k: v.to(model.device) for k, v in inputs.items()}
    
    with torch.no_grad():
        outputs = model(**inputs)
        embeddings = outputs.last_hidden_state[:, 0, :]  # CLS token
    
    # Normalize embeddings
    embeddings = torch.nn.functional.normalize(embeddings, p=2, dim=1)
    return embeddings.cpu().numpy()

texts = ["Hà Nội là thủ đô của Việt Nam", "Paris is the capital of France"]
embeddings = embed(texts)
print(embeddings.shape)  # (2, embedding_dim)

With vLLM (Recommended for serving)

bash

docker run -d \
  --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model dinhhungitsoft/Qwen3-Embedding-4B-bnb4 \
  --trust-remote-code \
  --gpu-memory-utilization 0.6 \
  --max-model-len 4096

Then query via OpenAI-compatible API:

python

import requests

response = requests.post(
    "http://localhost:8000/v1/embeddings",
    json={
        "model": "dinhhungitsoft/Qwen3-Embedding-4B-bnb4",
        "input": ["Your text here"]
    }
)
embeddings = response.json()["data"][0]["embedding"]

Languages

Optimized for:

English
Chinese
Vietnamese
And other languages supported by Qwen3

Hardware Requirements

GPU Memory: ~3-4GB VRAM (vs ~8GB for full precision)
Recommended: NVIDIA GPU with CUDA support
Tested on: RTX 3050 8GB

Limitations

Quantization may result in minor accuracy loss compared to full precision
Requires BitsAndBytes library for loading
Best performance with NVIDIA GPUs

Citation

If you use this model, please cite the original Qwen3 model:

bibtex

@article{qwen3,
  title={Qwen3 Technical Report},
  author={Qwen Team},
  year={2024}
}

License

Apache 2.0 (inherited from base model)