from vllm import LLM, SamplingParams

model = LLM("nm-testing/Nous-Hermes-2-Yi-34B-pruned50", sparsity="sparse_w16a16")
prompt = "How to make banana bread?"
formatted_prompt =  f"<|im_start|>User:{prompt}\n<|im_start|>assistant:\n"

sampling_params = SamplingParams(max_tokens=100, temperature=0)
outputs = model.generate(formatted_prompt, sampling_params=sampling_params)
print(outputs[0].outputs[0].text)
"""
To make banana bread, you will need the following ingredients:

Ingredients:
- 2 ripe bananas
- 1 cup all-purpose flour
- 1/2 cup sugar
- 1/2 cup butter
- 1 teaspoon baking soda
- 1 teaspoon baking powder
- 1/2 teaspoon salt
- 1/2 cup milk
- 1 teaspoon vanilla extract

Instructions:
1. Preheat the oven to 3
"""

Prompt template

<|im_start|>User:{prompt}
<|im_start|>assistant:

Sparsification

For details on how this model was sparsified, see the recipe.yaml in this repo and follow the instructions below.

Install SparseML:

bash

git clone https://github.com/neuralmagic/sparseml
pip install -e "sparseml[transformers]"

Replace the recipe as you like and run this one-shot compression script to apply SparseGPT:

python

import sparseml.transformers

original_model_name = "NousResearch/Nous-Hermes-2-Yi-34B"
calibration_dataset = "open_platypus"
output_directory = "output/"

recipe = """
test_stage:
  obcq_modifiers:
    SparseGPTModifier:
      sparsity: 0.5
      sequential_update: true
      mask_structure: 0:0
      targets: ['re:model.layers.\d*$']
"""

# Apply SparseGPT to the model
sparseml.transformers.oneshot(
    model=original_model_name,
    dataset=calibration_dataset,
    recipe=recipe,
    output_dir=output_directory,
)

Slack

For further support, and discussions on these models and AI in general, join Neural Magic's Slack Community

RedHatAI/Nous-Hermes-2-Yi-34B-pruned50

Nous-Hermes-2-Yi-34B-pruned50

Inference

Prompt template

Sparsification

Slack

RedHatAI/
Nous-Hermes-2-Yi-34B-pruned50