While the original model is excellent, running it at full precision can be resource-intensive. This AWQ version allows for high-throughput inference on consumer hardware (such as single or dual RTX 3090s) while maintaining the model's specialized capabilities.

Methodology

This model was converted using the LLM Compressor library (the currently recommended method for vLLM).

Quantization Scheme: W4A16 (4-bit weights, 16-bit activations)
Group Size: 128
Calibration Dataset: open_platypus

Credit

All credit for the training and architecture of the original model goes to Evan Armstrong (Heralax). This repository merely provides the quantized weights to facilitate broader adoption and easier deployment within the vLLM ecosystem.

How to Use with vLLM

python

from vllm import LLM, SamplingParams

# Load the model
llm = LLM(
    model="bbarn4/Augmentoolkit-DataSpecialist-v0.1-AWQ",
    quantization="awq"
)

# Generate
prompts = ["Hello, my name is"]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(f"Prompt: {output.prompt!r}, Generated text: {output.outputs[0].text!r}")