GGUF quantizations of Qwen/Qwen3-Embedding-4B — the mid-tier sweet spot for 16 GB+ Macs. Higher retrieval quality than 0.6B; leaner RAM than 8B.
Part of BatiAI's on-device RAG stack for BatiFlow.
recommended default — Q8↔Q6 drift cos 0.998, indistinguishable on retrieval
Qwen3-Embedding-4B-Q8_0.gguf
Q8_0
~4.3 GB
maximum quality, ~25 % bigger disk
Why Q6 over Q8 as default? Measured drift 0.998 = cosine distance well below any retrieval noise floor. Saving ~1 GB matters on 16 GB Macs where every GB of free RAM helps. If you want maximum conservatism, pull :4b-q8.
Why no IQ3 / IQ4 for embedding? Unlike chat LLMs, embedding quality cascades into cosine-similarity drift at low bit-widths. Q6_K / Q8_0 are the safe range.
All stages PASS with comfortable margin. Q8_0 and Q6_K produce essentially identical retrieval quality on this testset, with Q6_K saving ~25 % on disk.
4B sits in the middle — meaningfully better separation than 0.6B, within striking distance of 8B, at roughly half the disk footprint of 8B. Recommended default for 16 GB+ Macs.
Matryoshka — runtime-configurable dimension
Qwen3-Embedding-4B outputs up to 2560 dimensions. BatiFlow RAG stack defaults to 1024 (quality / latency sweet spot per our tests). Truncate at read time — no re-embed needed:
python
1emb = get_embedding(text)# [2560]2emb_1024 = emb[:1024]# truncate for storage savings3import numpy as np
4emb_1024 = emb_1024 / np.linalg.norm(emb_1024)# re-normalize
user query
↓ [Qwen3-Embedding 4B] ← YOU ARE HERE (mid tier)
1024-dim vector
↓ vector DB (sqlite-vec / LanceDB)
top-K candidates
↓ [Qwen3-Reranker 4B or 8B]
top-3
↓ [Qwen3.6-35B-A3B chat LLM]
answer
Recommended Usage — query vs document
python
1# Query side2query ="Instruct: Given a document query, retrieve the most relevant chunk.\n" \
3"Query: "+ user_input
45# Document side — no instruction prefix6document = chunk_text
BatiFlow handles this automatically. For custom integrations, see Qwen3-Embedding usage.
