You can deploy this model and start earning money today!

geoffsee/
octen-embedding-0.6b-onnx-int8

Name: octen-embedding-0.6b-onnx-int8
Availability: InStock

sentence-transformers

Views

No views yet

Octen-Embedding-0.6B ONNX (INT8)

ONNX export of Octen/Octen-Embedding-0.6B for inference with ONNX Runtime. This repo contains only the INT8 variant for smaller downloads. Other variants: fp16, int8, int4.

Base model: Octen/Octen-Embedding-0.6B
Max sequence length: 512
Hidden size: 1024

Files

File	Description
`model.int8.onnx`	INT8 quantized, ~560 MB
`tokenizer/`	Same as base model

Usage

python

from huggingface_hub import hf_hub_download
import onnxruntime as ort
from transformers import AutoTokenizer

repo_id = "geoffsee/octen-embedding-0.6b-onnx-int8"
path = hf_hub_download(repo_id=repo_id, filename="model.int8.onnx")
tokenizer = AutoTokenizer.from_pretrained(repo_id)
session = ort.InferenceSession(path, providers=["CPUExecutionProvider"])
# ... encode and run ...