You can deploy this model and start earning money today!

RichardErkhov/
neuralmagic_-_phi-2-pruned50-gguf

Name: neuralmagic_-_phi-2-pruned50-gguf
Availability: InStock

Views

No views yet

Quantization made by Richard Erkhov.

Github

Discord

Request more models

phi-2-pruned50 - GGUF

Model creator: https://huggingface.co/neuralmagic/
Original model: https://huggingface.co/neuralmagic/phi-2-pruned50/

Name	Quant method	Size
phi-2-pruned50.Q2_K.gguf	Q2_K	1.03GB
phi-2-pruned50.Q3_K_S.gguf	Q3_K_S	1.16GB
phi-2-pruned50.Q3_K.gguf	Q3_K	1.33GB
phi-2-pruned50.Q3_K_M.gguf	Q3_K_M	1.33GB
phi-2-pruned50.Q3_K_L.gguf	Q3_K_L	1.47GB
phi-2-pruned50.IQ4_XS.gguf	IQ4_XS	1.43GB
phi-2-pruned50.Q4_0.gguf	Q4_0	1.49GB
phi-2-pruned50.IQ4_NL.gguf	IQ4_NL	1.5GB
phi-2-pruned50.Q4_K_S.gguf	Q4_K_S	1.51GB
phi-2-pruned50.Q4_K.gguf	Q4_K	1.62GB
phi-2-pruned50.Q4_K_M.gguf	Q4_K_M	1.62GB
phi-2-pruned50.Q4_1.gguf	Q4_1	1.65GB
phi-2-pruned50.Q5_0.gguf	Q5_0	1.8GB
phi-2-pruned50.Q5_K_S.gguf	Q5_K_S	1.8GB
phi-2-pruned50.Q5_K.gguf	Q5_K	1.87GB
phi-2-pruned50.Q5_K_M.gguf	Q5_K_M	1.87GB
phi-2-pruned50.Q5_1.gguf	Q5_1	1.95GB
phi-2-pruned50.Q6_K.gguf	Q6_K	2.13GB
phi-2-pruned50.Q8_0.gguf	Q8_0	2.75GB

Original model description:

base_model: microsoft/phi-2 inference: true model_type: phi quantized_by: mgoin tags:

nm-vllm
sparse

phi-2-pruned50

This repo contains model files for Phi 2 optimized for NM-vLLM, a high-throughput serving engine for compressed LLMs.

This model was pruned with SparseGPT, using SparseML.

Inference

Install NM-vLLM for fast inference and low memory-usage:

pip install nm-vllm[sparse]

Run in a Python pipeline for local inference:

python

from vllm import LLM, SamplingParams

# Create a sparse LLM
llm = LLM("nm-testing/phi-2-pruned50", sparsity="sparse_w16a16")

prompt = "Once upon a time, there was a little car named Beep."
# Create a sampling params object.
sampling_params = SamplingParams(temperature=0.0, max_tokens=200)

# Generate texts from the prompts. The output is a list of RequestOutput objects
# that contain the prompt, generated text, and other information.
outputs = llm.generate(prompt, sampling_params)
# Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"\nGenerated text: {prompt}{generated_text}\n")

"""
Generated text: Once upon a time, there was a little car named Beep. Beep was a small car, but he was very fast and loved to go on adventures. Beep had a friend named Bop who was a big car. Bop was very slow and loved to stay at home. Beep and Bop were very different, but they were still friends.

One day, Beep and Bop decided to go on an adventure together. Beep was excited to explore new places and Bop was excited to see Beep explore. They started their adventure by driving on a bumpy road. Beep was having a great time, but Bop was having a hard time. Bop was so big that he couldn't fit in the small spaces between the bumps. Beep was having a great time, but Bop was having a hard time.

As they continued their adventure, they came across a big hill. Beep was excited to climb the hill, but Bop was scared. Bop was so big that he couldn't
"""

Prompt template

"Instruct: <prompt>\nOutput:"

Sparsification

For details on how this model was sparsified, see the recipe.yaml in this repo and follow the instructions below.

Install SparseML:

bash

git clone https://github.com/neuralmagic/sparseml
pip install -e "sparseml[transformers]"

Replace the recipe as you like and run this one-shot compression script to apply SparseGPT: