This model is an HF optimum 0.0.28 (AWS Neuron SDK 2.20.2)'s compiled verson, of the Korean fine-tuned model Qwen/Qwen2.5-7B-Instruct , available at https://huggingface.co/Qwen/Qwen2.5-7B-Instruct. It is intended for deployment on Amazon EC2 Inferentia2 and Amazon SageMaker.

This model is compiled with HF optimum 0.0.28, neuronx-cc version: 2.15.143 v1.2-hf-tgi-0.0.28-pt-2.1.2-inf-neuronx-py310 Please refer to a guide at https://github.com/aws-samples/aws-ai-ml-workshop-kr/tree/master/neuron/hf-optimum/04-Deploy-Qwen-25-8B-Llama3-8B-HF-TGI-Docker-On-INF2

At a minimum hardware, you can use Amazon EC2 inf2.xlarge and more powerful family such as inf2.8xlarge, inf2.24xlarge and inf2.48xlarge and them at SageMaker Inference endpoing. The detailed information is Amazon EC2 Inf2 Instances

Gonsoo Moon, gonsoomoon@gmail.com