The quantised model's configs and weights are stored in hf and safetensors format, but the tokeniser remains in mistral format. Please load inference arguments accordingly, e.g.,:

vllm

vllm serve cpatonn/Devstral-Small-2507-AWQ --tokenizer_mode mistral --config_format hf --load_format safetensors --tool-call-parser mistral --enable-auto-tool-choice

btbtyler09/Devstral-Small-2507-AWQ

Devstral-Small-2507-AWQ

Method

Inference

vllm

btbtyler09/
Devstral-Small-2507-AWQ