This model was quantized with GPTQ and saved in the Marlin format for efficient 4-bit inference. Marlin is a highly optimized inference kernel for 4 bit models.

Inference

Install nm-vllm for fast inference and low memory-usage:

pip install nm-vllm[sparse]

Run in a Python pipeline for local inference:

python

from transformers import AutoTokenizer
from vllm import LLM, SamplingParams

model_id = "neuralmagic/TinyLlama-1.1B-Chat-v1.0-marlin"
model = LLM(model_id)

tokenizer = AutoTokenizer.from_pretrained(model_id)
messages = [
    {"role": "user", "content": "How to make banana bread?"},
]
formatted_prompt =  tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
sampling_params = SamplingParams(max_tokens=200)
outputs = model.generate(formatted_prompt, sampling_params=sampling_params)
print(outputs[0].outputs[0].text)

"""
Sure! Here's a simple recipe for banana bread:

Ingredients:
- 3-4 ripe bananas,mashed
- 1 large egg
- 2 Tbsp. Flour
- 2 tsp. Baking powder
- 1 tsp. Baking soda
- 1/2 tsp. Ground cinnamon
- 1/4 tsp. Salt
- 1/2 cup butter, melted
- 3 Cups All-purpose flour
- 1/2 tsp. Ground cinnamon

Instructions:

1. Preheat your oven to 350 F (175 C).
"""

Quantization

For details on how this model was quantized and converted to marlin format, run the quantization/apply_gptq_save_marlin.py script:

bash

pip install -r quantization/requirements.txt
python3 quantization/apply_gptq_save_marlin.py --model-id TinyLlama/TinyLlama-1.1B-Chat-v1.0 --save-dir ./tinyllama-marlin

Slack

For further support, and discussions on these models and AI in general, join Neural Magic's Slack Community

RedHatAI/TinyLlama-1.1B-Chat-v1.0-marlin

TinyLlama-1.1B-Chat-v1.0

Inference

Quantization

Slack

RedHatAI/
TinyLlama-1.1B-Chat-v1.0-marlin