yujiepan/

mpt-7b-gsm8k-dummy

pip install deepsparse-nightly[llm]==1.6.0.20231120
pip install openvino==2023.3.0

Benchmarking

1. Clone this repo
2. Concatenate the big fp32 IR model:

1. Reproduce NM paper: deepsparse_reproduce.bash
2. OV benchmarkapp: benchmarkapp_*.bash

Generating these IRs