Views
No views yet
Qwen/Qwen3-0.6Btext-generation17FP32 (No Quantization)onnxruntime and tokenizers.pip install onnxruntime tokenizers optimum1
2from tokenizers import Tokenizer
3import onnxruntime as ort
4import numpy as np
5
6# 1. Load the lightweight tokenizer (No Transformers dependency needed)
7tokenizer = Tokenizer.from_pretrained("broadfield-dev/Qwen3-0.6B-onnx")
8
9# 2. Load the ONNX model
10# For Generative/Chat models, use: optimum.onnxruntime.ORTModelForCausalLM
11session = ort.InferenceSession("model.onnx")
12
13# 3. Preprocess (Simple text encoding)
14text = "Run inference on mobile!"
15encoding = tokenizer.encode(text)
16
17# Prepare inputs (Exact names vary by model, usually input_ids + attention_mask)
18inputs = {
19 "input_ids": np.array([encoding.ids], dtype=np.int64),
20 "attention_mask": np.array([encoding.attention_mask], dtype=np.int64)
21}
22
23# 4. Run Inference
24outputs = session.run(None, inputs)
25print("Output logits shape:", outputs[0].shape)
26FP32 (No Quantization) quantization settings and a pre-compiled tokenizer.json for fast loading.