Property	Value
Original Model	voyageai/voyage-4-nano
Format	ONNX (Int8 Dynamic Quantization)
Model Size	~329 MB (53% smaller than FP16)
Dimensions	1024 (default), 512, 256 via Matryoshka
Context Length	32,768 tokens
Quality vs FP16	~97% cosine similarity
License	Apache 2.0

Why Int8?

Smaller model size: 329 MB vs 705 MB (FP16) - 53% reduction
Faster CPU inference: Int8 weights are faster to process on CPU
Maintained quality: ~97% cosine similarity with FP16 model
Perfect for edge/mobile: Smaller download, lower memory usage

voyage-4-nano was trained with Quantization-Aware Training (QAT), making it well-suited for int8 quantization with minimal quality loss.

Usage

Transformers.js (Browser/Node.js)

javascript

import { pipeline } from '@huggingface/transformers';

const extractor = await pipeline(
  'feature-extraction',
  'YOUR_USERNAME/voyage-4-nano-ONNX-int8'
);

// Document embedding (for indexing)
const docPrefix = "Represent the document for retrieval: ";
const docEmbedding = await extractor(docPrefix + "Your document text", {
  pooling: 'mean',
  normalize: true
});

// Query embedding (for search)
const queryPrefix = "Represent the query for retrieving supporting documents: ";
const queryEmbedding = await extractor(queryPrefix + "Your search query", {
  pooling: 'mean',
  normalize: true
});

ONNX Runtime (Python)

python

import onnxruntime as ort
from transformers import AutoTokenizer
import numpy as np

tokenizer = AutoTokenizer.from_pretrained("YOUR_USERNAME/voyage-4-nano-ONNX-int8")
session = ort.InferenceSession("model.onnx")

text = "Represent the document for retrieval: Your text here"
inputs = tokenizer(text, return_tensors="np", padding=True, truncation=True)

# Create position_ids
seq_len = inputs["input_ids"].shape[1]
position_ids = np.arange(seq_len).reshape(1, -1).astype(np.int64)

outputs = session.run(None, {
    "input_ids": inputs["input_ids"],
    "attention_mask": inputs["attention_mask"],
    "position_ids": position_ids
})

# Mean pooling
embeddings = outputs[0]
mask = inputs["attention_mask"]
pooled = (embeddings * mask[:, :, None]).sum(1) / mask.sum(1, keepdims=True)

# L2 normalize
normalized = pooled / np.linalg.norm(pooled, axis=1, keepdims=True)

Instruction Prefixes

Important: Use the appropriate prefix for best results:

Documents (indexing): "Represent the document for retrieval: "
Queries (search): "Represent the query for retrieving supporting documents: "

Quantization Details

This model was quantized using ONNX Runtime's dynamic quantization:

Quantization type: Dynamic (weights quantized to int8, activations remain float32)
Source model: FP16 ONNX (converted to FP32 intermediate for quantization)
Target operators: MatMul, Attention layers
Quality retention: ~97% cosine similarity vs FP16 original

When to use Int8 vs FP16

Use Case	Recommended
Production server with GPU	FP16
Edge/mobile deployment	Int8
Browser-based inference	Int8
Bandwidth-constrained	Int8
Maximum precision needed	FP16

Model Comparison

Model	Size	Quality	Use Case
voyage-4-nano-ONNX (FP16)	705 MB	Baseline	Production, high precision
voyage-4-nano-ONNX-int8	329 MB	~97%	Edge, mobile, browser

Validation Results

All validation tests pass:

Model loads and runs correctly
Embedding dimensions: 1024
Cosine similarity vs FP16: ~0.97 (average)
Semantic ranking preserved
L2 normalization correct
Transformers.js compatible

License

Apache 2.0 (same as original model)

Acknowledgments

Voyage AI for the original model
Hugging Face for Transformers.js and ONNX Runtime

jsonMartin/voyage-4-nano-ONNX-int8

voyage-4-nano-ONNX-int8

Model Details