You can deploy this model and start earning money today!

12Sanju21/
jina-embedder-v4-nano-for-vespa

Name: jina-embedder-v4-nano-for-vespa
Availability: InStock

onnx

apache-2.0

Views

No views yet

Deploying ONNX Embedder on GPU

This guide explains how to set up a REST API endpoint to serve text embeddings using an ONNX model and a local tokenizer, accelerated by NVIDIA GPUs.

1. Prerequisites

Ensure your environment has the following installed:

CUDA Toolkit (compatible with your GPU)
cuDNN
Python 3.9+

2. Environment Setup

Install the necessary dependencies. The onnxruntime-gpu package is required to leverage the CUDAExecutionProvider.

pip install fastapi uvicorn onnxruntime-gpu transformers torch

3. Implementation Logic

To create the endpoint, we need to:

Initialize the Tokenizer.
Load the ONNX Session with the GPU provider.
Process the input text into tensors.
Run the inference and return the embeddings.

4. Deployment Script (`app.py`)

You can use the following boilerplate for your API:

python

import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

# 1. Load Tokenizer and ONNX Session
tokenizer = AutoTokenizer.from_pretrained("./tokenizer_path")
session = ort.InferenceSession(
    "model.onnx", 
    providers=['CUDAExecutionProvider', 'CPUExecutionProvider']
)

class Query(BaseModel):
    text: str

@app.post("/embed")
async def get_embeddings(query: Query):
    # 2. Tokenization
    inputs = tokenizer(query.text, return_tensors="np", padding=True, truncation=True)
    
    # 3. ONNX Inference
    # Ensure input names match your specific ONNX model (usually 'input_ids', 'attention_mask')
    onnx_inputs = {node.name: inputs[node.name] for node in session.get_inputs()}
    outputs = session.run(None, onnx_inputs)
    
    # 4. Return result (usually the first output)
    embeddings = outputs[0].tolist()
    return {"embeddings": embeddings}

5. Running the Server

Start the API using Uvicorn:

uvicorn app:app --host 0.0.0.0 --port 8000

6. Making Requests

You can now send requests to your GPU-accelerated endpoint:

bash

curl -X POST "http://localhost:8000/embed" \
  -H "Content-Type: application/json" \
  -d @- <<EOFEOF
  {
    "model": "jina-embeddings-v4",
    "task": "text-matching",
    "input": [
        {
            "text": "A beautiful sunset over the beach"
        },
        {
            "text": "Un beau coucher de soleil sur la plage"
        },
        {
            "text": "Ein wunderschöner Sonnenuntergang am Strand"
        },
        {
            "text": "Ένα όμορφο ηλιοβασίλεμα πάνω από την παραλία"
        },
        {
            "text": "Un bellissimo tramonto sulla spiaggia"
        },
        {
            "image": "https://i.ibb.co/nQNGqL0/beach1.jpg"
        },
        {
            "image": "https://i.ibb.co/r5w8hG8/beach2.jpg"
        }
    ]
  }
EOFEOF

Key Technical Notes for your README:

Execution Providers: In the InferenceSession, placing 'CUDAExecutionProvider' first is critical. If the GPU drivers are missing, it will gracefully fall back to 'CPUExecutionProvider'.
Input Names: Remind users to check their ONNX model input names using [node.name for node in session.get_inputs()], as some models use token_type_ids while others do not.
Optimization: For even higher throughput, mention using Fixed-shape inputs or FP16 quantization to further reduce GPU memory usage.

12Sanju21/jina-embedder-v4-nano-for-vespa