name: qwen3-embedding
backend: llama-cpp
files:
  - filename: Qwen3-Embedding-4B-TQ4_1S.gguf
    uri: huggingface://grevinden/Qwen3-Embedding-4B-TQ4_1S-GGUF/Qwen3-Embedding-4B-TQ4_1S.gguf

После этого перезапустить LocalAI — модель подтянется автоматически.

OpenAI-compatible API

Original model: <a href="https://huggingface.co/Qwen/Qwen3-Embedding-4B" target="_blank" rel="noopener noreferrer" class="text-blue-400 hover:text-blue-300 underline underline-offset-2 transition-colors">Qwen/Qwen3-Embedding-4B
Quantization: TQ4_1S (TurboQuant, ~5.0 bpw, WHT-rotated)
Size: ~2.5 GB (from 7.5 GB f16)
GGUF backend: llama.cpp TQP v0.2.0 (TurboQuant Plus)

bash

curl http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "input": "Hello world",
    "model": "qwen3-embedding"
  }'

HuggingFace Hub (llama-cpp-python)

python

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="grevinden/Qwen3-Embedding-4B-TQ4_1S-GGUF",
    filename="Qwen3-Embedding-4B-TQ4_1S.gguf",
    embedding=True,
)
embeddings = llm.create_embedding("Hello world")

Характеристики модели

Параметр	Значение
Hidden size	2560
Layers	35
Attention heads	20
Max tokens	32768
Embedding dim	2560
Pooling	cls
GGUF type	TQ4_1S (5.15 BPW)
Размер файла	~2.5 GB