File	Quant	Size	Description
`Octen-Embedding-8B-Q4_K_M.gguf`	Q4_K_M	4.0 GB	Good balance of size and quality
`Octen-Embedding-8B-Q6_K.gguf`	Q6_K	6.5 GB	High quality, moderate size
`Octen-Embedding-8B-Q8_0.gguf`	Q8_0	8.0 GB	Near-lossless, recommended

All quantizations were created with --leave-output-tensor and --token-embedding-type F16 to preserve embedding quality.

Usage with llama.cpp

bash

llama-embedding \
  -m Octen-Embedding-8B-Q8_0.gguf \
  --pooling last \
  -p "Your text here"

Usage with llama-cpp-python

python

from llama_cpp import Llama

llm = Llama(
    model_path="Octen-Embedding-8B-Q8_0.gguf",
    embedding=True,
    n_gpu_layers=-1,
    n_ctx=2048,
)

result = llm.create_embedding("Your text here")
embedding = result['data'][0]['embedding']  # 4096-dim vector

Conversion Command

bash

# Step 1: Convert to F16
python convert_hf_to_gguf.py Octen/Octen-Embedding-8B \
  --outfile Octen-Embedding-8B-f16.gguf \
  --outtype f16

# Step 2: Quantize
llama-quantize \
  --leave-output-tensor \
  --token-embedding-type F16 \
  Octen-Embedding-8B-f16.gguf \
  Octen-Embedding-8B-Q8_0.gguf Q8_0

tex8/Octen-Embedding-8B-GGUF

Octen-Embedding-8B - GGUF Quantizations

Available Quantizations

Usage with llama.cpp

Usage with llama-cpp-python

Conversion Command

tex8/
Octen-Embedding-8B-GGUF