Quantization made by Richard Erkhov.
base_model: neuralmagic/Llama-2-7b-pruned70-retrained
inference: true
model_type: llama
pipeline_tag: text-generation
datasets:
This repo contains a
70% sparse Llama 2 7B finetuned for instruction-following tasks using a blend of the Platypus + Open Orca + Dolphin datasets.
Below we share some code snippets on how to get quickly started with running the model.
By leveraging a pre-sparsified model's structure, you can efficiently fine-tune on new data, leading to reduced hyperparameter tuning, training times, and computational costs. Learn about this process
here.
This model may be run with the transformers library. For accelerated inference with sparsity, deploy with
nm-vllm or
deepsparse.
1# pip install transformers accelerate
2from transformers import AutoTokenizer, AutoModelForCausalLM
3
4tokenizer = AutoTokenizer.from_pretrained("Llama-2-7b-pruned70-retrained-instruct")
5model = AutoModelForCausalLM.from_pretrained("Llama-2-7b-pruned70-retrained-instruct", device_map="auto")
6
7input_text = "Write a recipe for banana bread:\n"
8input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
9
10outputs = model.generate(**input_ids)
11print(tokenizer.decode(outputs[0]))
Model evaluation metrics and results.
This model was obtained by sparse-tranfer of the sparse foundational model
Llama-2-7b-pruned50-retrained on a blend of
Open Platypus, 10%
Open Orca and 10%
Dolphin datasets. Training was perfomerd for 6 epochs.
For further support, and discussions on these models and AI in general, join
Neural Magic's Slack Community
