Views
No views yet
mistralai/Ministral-3-3B-Instruct-2512.
Language-model linears are compressed-tensors W4A16; the Pixtral vision tower and multimodal projector remain BF16.1docker run --rm -it \
2 --gpus all \
3 --ipc=host \
4 -p 8001:8000 \
5 -v ~/.cache/huggingface:/root/.cache/huggingface \
6 vllm/vllm-openai:latest \
7 mistralai/Ministral-3-3B-Instruct-2512 \
8 --served-model-name Ministral-3-3B-Instruct-2512-stock \
9 --dtype bfloat16 \
10 --max-model-len 8192 \
11 --gpu-memory-utilization 0.71docker run --rm -it \
2 --gpus all \
3 --ipc=host \
4 -p 8002:8000 \
5 -v /path/to/Ministral-3-3B-Instruct-2512-W4A16-BF16Vision:/model \
6 -v ~/.cache/huggingface:/root/.cache/huggingface \
7 vllm/vllm-openai:latest \
8 /model \
9 --served-model-name Ministral-3-3B-Instruct-2512-W4A16-BF16Vision \
10 --dtype bfloat16 \
11 --quantization compressed-tensors \
12 --max-model-len 8192 \
13 --gpu-memory-utilization 0.7python verify.py --url http://localhost:8002/v1/chat/completions