Same int8 weight-only group_size-128 quantization as Qwen3-Embedding-0.6B-W8A16-g128, but packaged as repo custom code (trust_remote_code=True) whose Int8GroupLinear modules keep the int8 weights RESIDENT in VRAM and dequantize one layer at a time inside forward.

Why: transformers decompresses compressed-tensors checkpoints to bf16 in VRAM (lazily, at first forward) and keeps the bf16, so the stock format saves download but not resident memory. This packaging saves both.

Measured (H100):

	bf16	this repo
Resident weights (transformers, both after forwards)	1136 MiB	740 MiB
Infinity 0.0.77, `--batch-size 8`, process VRAM	2490 MiB	2102 MiB (-16%)
Latency in-process, batch 1 / batch 32	17.2 / 23.3 ms	23.0 / 29.8 ms

Fidelity: quantization identical to the W8A16-g128 repo (recall@10 0.976 vs bf16 on a 360-doc multi-script index; embedding cosine 0.999 vs the bf16 service).

Usage

python

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("Nicolassuez/Qwen3-Embedding-0.6B-W8A16-g128-resident",
                            trust_remote_code=True)

or AutoModel.from_pretrained(..., trust_remote_code=True) with last-token pooling + L2 normalize. Works in Infinity:

infinity_emb v2 --model-id Nicolassuez/Qwen3-Embedding-0.6B-W8A16-g128-resident \
    --trust-remote-code --no-bettertransformer --device cuda

(--no-bettertransformer is required: Infinity 0.0.77's BetterTransformer probe crashes when optimum is absent or >= 2.0.)

nn.Module.to(dtype) only casts floating-point tensors, so the int8 buffers survive the .to(float16/bfloat16) calls sentence-transformers and Infinity make.

Caveat: transformers version pinning

Qwen3 forward numerics drift between transformers 4.57 and 5.14 (embedding cosine ~0.973 between the two, measured on this model; the drift exists for the bf16 base too). Embeddings indexed with one version should be queried with the same version.

Update: int8 embedding table

The embedding table (151669 x 1024, 26% of parameters) is now ALSO stored int8, with one bf16 scale per row (a lookup gathers single rows, no accumulation crosses the quantized axis, so per-row scaling is the natural granularity). Resident weights drop 740 -> 592 MiB. Validated on the 360-doc / 70-query benchmark against bf16: mean weight cosine unchanged (0.99922 vs 0.99923 with the bf16 table) and every top-1 change happens on candidate pairs the bf16 model itself separates by less than 0.01. Do NOT replicate this on the reranker: its lm_head is tied to the table and the yes/no score reads it directly.