You can deploy this model and start earning money today!

geoffsee/
octen-embedding-0.6b-onnx-int4

Name: octen-embedding-0.6b-onnx-int4
Availability: InStock

sentence-transformers

Views

No views yet

Octen-Embedding-0.6B ONNX (INT4)

ONNX export of Octen/Octen-Embedding-0.6B for inference with ONNX Runtime. This repo contains only the INT4 variant for smaller downloads. Other variants: fp16, int8, int4.

Base model: Octen/Octen-Embedding-0.6B
Max sequence length: 512
Hidden size: 1024

Files

File	Description
`model.int4.onnx` (+ `.onnx.data`)	INT4 Bnb4 weight-only, ~537 MB
`tokenizer/`	Same as base model

Usage

python

from huggingface_hub import hf_hub_download
import onnxruntime as ort
from transformers import AutoTokenizer

repo_id = "geoffsee/octen-embedding-0.6b-onnx-int4"
path = hf_hub_download(repo_id=repo_id, filename="model.int4.onnx")
tokenizer = AutoTokenizer.from_pretrained(repo_id)
session = ort.InferenceSession(path, providers=["CPUExecutionProvider"])
# ... encode and run ...