Views
No views yet
neuron format using specific input_shapes and compiler parameters detailed in the paragraphs below.optimum-neuron documentation for an explanation of these parameters.optimum-neuron1>>> from optimum.neuron import pipeline
2
3>>> p = pipeline('text-generation', 'davidshtian/Mistral-7B-Instruct-v0.2-neuron-1x2048-2-cores')
4>>> p("My favorite place on earth is", max_new_tokens=64, do_sample=True, top_k=50)
5[{'generated_text': 'My favorite place on earth is Hawaii,” she said, her voice bright and clear despite her quietness.
6“That place, and the ocean. But it’s hard to ever live there permanently. The ocean is there and it calls to me, but
7it’s big and vast and doesn’t allow me a lot of freedom.”'}]neuronx. When using with 🤗 optimum-neuron, use the repo revision specific to the version of neuronx you are using, to load the right serialized checkpoints.1{
2 "batch_size": 1,
3 "sequence_length": 2048,
4}1{
2 "auto_cast_type": "bf16",
3 "num_cores": 2,
4}