from transformers import AutoModel, AutoTokenizer

# Load quantized model
model = AutoModel.from_pretrained(
    "nomic-embed-code-W4A16-AWQ",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    "nomic-embed-code-W4A16-AWQ",
    trust_remote_code=True
)

# Generate embeddings
texts = ["Hello world", "Example text"]
inputs = tokenizer(texts, padding=True, return_tensors="pt")
embeddings = model(**inputs).last_hidden_state.mean(dim=1)

print(embeddings.shape)

Performance

Memory usage: ~75% reduction vs FP16
Inference speed: Similar or faster on compatible hardware
Quality: Minimal degradation (<1% on most embedding tasks)

Why AWQ?

AWQ (Activation-aware Weight Quantization) is a one-shot weight quantization method that:

Activation-aware: Protects salient weights based on activation magnitudes
Uses calibration data to identify important weight channels
Provides better accuracy than GPTQ and naive rounding (RTN)
Works efficiently with group-wise quantization (group size 128)
Maintains model quality while achieving 75% size reduction
Optimal for embedding models that rely on preserving semantic relationships

Original Model

This quantized model is based on nomic-ai/nomic-embed-code.

Citation

If you use this model, please cite the original model and llmcompressor:

bibtex

@software{llmcompressor,
  title = {LLM Compressor},
  author = {Neural Magic},
  url = {https://github.com/vllm-project/llm-compressor},
  year = {2024}
}

pyrymikko/nomic-embed-code-W4A16-AWQ

nomic-embed-code-W4A16-AWQ

Quantization Details

Usage

Performance

Why AWQ?

Original Model

Citation

pyrymikko/
nomic-embed-code-W4A16-AWQ