from vllm import LLM

model = LLM("edge-inference/DSR1-32B-llmc-awq-w4")
output = model.generate("Hello, how are you?")
print(output[0].outputs[0].text)

Transformers

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("edge-inference/DSR1-32B-llmc-awq-w4")
tokenizer = AutoTokenizer.from_pretrained("edge-inference/DSR1-32B-llmc-awq-w4")

Performance

Memory: 72% reduction (18GB vs 64GB)
Speed: Faster inference due to reduced memory bandwidth
Quality: Minimal degradation with AWQ quantization

License

Same as base model (DeepSeek License)