Views
No views yet
| Folder | Precision | EP / Device |
|---|---|---|
f16/cuda | FP16 | CUDA |
f16/default | FP16 | Default (CPU / DML / WebGPU) |
bf16/cuda | BF16 | CUDA |
bf16/default | BF16 | Default (DML / WebGPU; bf16 MatMul is not on the CPU EP) |
Q4_K_M/cuda | INT4 (K-Quant Q4_K_M) | CUDA |
Q4_K_M/default | INT4 (K-Quant Q4_K_M) | Default (CPU) |
Q4_RTN/cuda | INT4 (RTN; quantized embeddings + LM head + body) | CUDA |
Q4_GPTQ/cuda | INT4 (GPTQ body + RTN embeddings / LM head, tied embeddings) | CUDA |
Q4_KQuant_tie/cuda | INT4 (K-Quant body + RTN embeddings / LM head, tied embeddings) | CUDA |
Q4_KQuant_tie/cpu | INT4 (K-Quant body + RTN embeddings / LM head, tied embeddings) | CPU |
Q4_KQuant_tie/webgpu | INT4 (K-Quant body + RTN embeddings / LM head, tied embeddings) | WebGPU |
*/cuda packages set the CUDA execution provider in genai_config.json;
*/default packages leave provider_options empty so ORT GenAI runs them on
whatever EP is configured (CPU by default).Q4_RTN/cuda is fully weight-quantized: in addition to the body projections, the input embedding table (GatherBlockQuantized) and the tied LM head (MatMulNBits) are INT4, giving the smallest footprint.Q4_GPTQ/cuda quantizes the same layers to INT4 (group size 32) but uses
GPTQ (Hessian-aware, wikitext-calibrated) for the transformer body instead
of RTN, with RTN only for the embedding table and LM head. The input embedding
and LM head share a single tied INT4 table, so it is both more accurate
(GPTQ) and ~11% smaller on disk (~1.03 GiB vs ~1.16 GiB) than Q4_RTN/cuda.
Recommended INT4 CUDA variant.Q4_KQuant_tie/* uses K-Quant (Q4_K_M) for the transformer body and RTN for
the input embedding table and LM head, which share a single tied INT4 table.
It has the best quality/size trade-off of the INT4 variants (smallest footprint,
highest top-1 agreement with FP16). cuda and webgpu are byte-for-byte identical
(fp16 scales, ~1.03 GiB); cpu stores fp32 scales (~1.19 GiB) since the CPU EP
prefers fp32.model.onnx, model.onnx.data,
genai_config.json, tokenizer, chat template).1import onnxruntime_genai as og
2from transformers import AutoTokenizer
3
4path = "Q4_K_M/default" # or f16/cuda, bf16/cuda, Q4_K_M/cuda
5model = og.Model(path)
6tok = AutoTokenizer.from_pretrained("tencent/Hy-MT2-1.8B")
7
8src, tgt = "黄河之水天上来", "English"
9messages = [{"role": "user",
10 "content": f"Translate the following text into {tgt}:\n{src}"}]
11prompt = tok.apply_chat_template(messages, tokenize=False,
12 add_generation_prompt=True)
13ids = tok(prompt, return_tensors="np")["input_ids"].astype("int32")
14
15params = og.GeneratorParams(model)
16params.set_search_options(max_length=256, do_sample=False)
17gen = og.Generator(model, params)
18gen.append_tokens(ids[0])
19out = []
20while not gen.is_done():
21 gen.generate_next_token()
22 out.append(gen.get_next_tokens()[0])
23print(tok.decode(out, skip_special_tokens=True))
24# -> The water of the Yellow River comes from the skyThe Hy-MT BPE vocab uses a custom regex pre-tokenizer that ort-extensions does not currently round-trip, so use the HuggingFace tokenizer to encode / decode and feed raw token IDs toog.Generator.
tencent-Hy-MT2-1.8B): MobiusBuilder for the FP16/BF16 exports,
MobiusBuilder → OnnxKQuantQuantization for the INT4 K-Quant variants,
rtn → MobiusBuilder for Q4_RTN, gptq → rtn → MobiusBuilder
for Q4_GPTQ, and MobiusBuilder → OnnxKQuantQuantization (body) →
OnnxBlockWiseRtnQuantization (embeddings) → GraphSurgeries[TieWordEmbeddings]
for Q4_KQuant_tie.