This repository hosts the optimized versions of Llama-3.2-3B-Instruct to accelerate inference with ONNX Runtime.
Optimized models are published here in ONNX format to run with ONNX Runtime on CPU and GPU across devices, including server platforms, Windows, Linux and Mac desktops, and mobile CPUs, with the precision best suited to each of these targets.
To easily get started with the model, you can use our newly introduced ONNX Runtime Generate() API. See here for instructions on how to run it.
ONNX Models
Here are some of the optimized configurations we have added:
ONNX model for int4 CPU and Mobile: ONNX model for CPU and mobile using int4 quantization via RTN
ONNX model for int4 GPU using quantization via RTN.
CPU SKU: Standard D16s v6 (16 vcpus, 64 GiB memory)
AMD CPU: Internal_D64as_v5
Minimum Configuration Required:
Windows: DirectX 12-capable GPU and a minimum of 4GB of combined RAM
CUDA: NVIDIA GPU with Compute Capability >= 7.0
Model Description
Developed by: ONNX Runtime, Microsoft
Model type: ONNX
Language(s) (NLP): Python, C, C++
License: MIT
License: Use of Llama 3.2 is governed by the Llama 3.2 Community License (a custom, commercial license agreement).
Model Description: This is a conversion of the Llama 3.2 model for ONNX Runtime inference.
Disclaimer: Model is only an optimization of the base model, any risk associated with the model is the responsibility of the user of the model. Please verify and test for you scenarios. There may be a slight difference in output from the base model with the optimizations applied. **
How to Get Started with the Model
To make running of the models across a range of devices and platforms across various execution provider backends possible, we introduce a new API to wrap several aspects of generative AI inferencing. This API make it easy to drag and drop LLMs straight into your app. For running the early version of these models with ONNX Runtime, follow the steps here.
*Input:* <|user|>Tell me a joke<|end|><|assistant|>
*Output:* Why don't scientists trust atoms?
Because they make up everything!
This joke plays on the double meaning of "make up." In science, atoms are the fundamental building blocks of matter, literally making up everything. However, in a colloquial sense, "to make up" can mean to fabricate or lie, hence the humor.
Model performance
Up to 1.25X faster than llama.cpp on Standard F16s v2 (16 vcpus, 32 GiB memory)
Up to 39X faster than PyTorch compile on Standard_ND96amsr_A100_v4.
Base Model Information
"See Meta's model card
Llama-3.2-3B-Instruct
for more information about the base model, including the base model's specific approach to responsible AI risks"
