from vllm import LLM, SamplingParams

# Initialize the model
llm = LLM(model="Vykyan/KAT-Dev-72B-Exp-GPTQ-INT4-gs128", trust_remote_code=True)

# Create sampling parameters
sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=512,
)

# Generate text
prompts = ["Hello, how are you?"]
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(output.outputs[0].text)

With Transformers

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Vykyan/KAT-Dev-72B-Exp-GPTQ-INT4-gs128",
    device_map="auto",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    "Vykyan/KAT-Dev-72B-Exp-GPTQ-INT4-gs128",
    trust_remote_code=True
)

# Generate
inputs = tokenizer("Hello, how are you?", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Inference Performance

This quantized model offers:

~4x memory reduction compared to FP16
Faster inference on compatible hardware
Maintained accuracy through GPTQ quantization

Recommended Hardware

NVIDIA GPUs with compute capability 7.5+ (RTX 20-series or newer)
Minimum 24GB VRAM for single-GPU inference
Multi-GPU setup for larger batch sizes

Model Details

Base Model: Kwaipilot/KAT-Dev-72B-Exp
Quantization Tool: llm-compressor
Compatible Inference Engines: vLLM, TGI (Text Generation Inference)

Limitations

Quantization may affect model accuracy on certain tasks
Requires vLLM or compatible inference engine for optimal performance

Acknowledgements

Base model: Kwaipilot/KAT-Dev-72B-Exp
Quantization: llm-compressor
Inference: vLLM

MidnightPhreaker/KAT-Dev-72B-Exp-GPTQ-INT4-gs128

KAT-Dev-72B-Exp - GPTQ INT4 (group_size=128)

Quantization Details

Hardware Used for Quantization

Usage

With vLLM

With Transformers

Inference Performance

Recommended Hardware

Model Details

Limitations

Acknowledgements

MidnightPhreaker/
KAT-Dev-72B-Exp-GPTQ-INT4-gs128