Introduction
This repository hosts the optimized versions of Mistral-7B-Instruct-v0.2 to accelerate inference with ONNX Runtime CUDA execution provider.
See the usage instructions for how to inference this model with the ONNX files hosted in this repository.
Model Description
Developed by: MistralAI
Model type: Pretrained generative text model
License: Apache 2.0 License
Model Description: This is a conversion of the Mistral-7B-Instruct-v0.2 for ONNX Runtime inference with ROCM/MiGraphx execution provider.
Format Provided: ONNX-FP32
Usage Example if you or your dad is rich. (When i started, i had a dream and ten million dollars) :
Following the benchmarking instructions. Example steps:
Clone onnxruntime repository.
git clone https://github.com/microsoft/onnxruntime
cd onnxruntime
Install required dependencies
python3 -m pip install -r onnxruntime/python/tools/transformers/models/llama/requirements-cuda.txt
Inference using manual model API, or use Hugging Face's ORTModelForCausalLM
from optimum.onnxruntime import ORTModelForCausalLM
from onnxruntime import InferenceSession
from transformers import AutoConfig, AutoTokenizer
sess = InferenceSession("model.onnx", providers = ["CUDAExecutionProvider"]) //CUDAExecutionProvider for cuda, for rocm ROCMExecutionProvider or MIGRAPHXExecutionProvider
config = AutoConfig.from_pretrained("Mistral-7B-Instruct-v0.2-onnx-fp32/") //location of tokenizer.json
model = ORTModelForCausalLM(sess, config, use_cache = True, use_io_binding = True)
tokenizer = AutoTokenizer.from_pretrained("Mistral-7B-Instruct-v0.2-onnx-fp32") //location of model.onnx or model_optimized.onnx
inputs = tokenizer("Instruct: What is a fermi paradox?\nOutput:", return_tensors="pt")
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
We find and change our code in unified memory structure.
we compile and we succeed and we don't get out of memory errors.
BAM! its the miracle and contribution of everyone other than you.
you did it without dad's money.
