Views
No views yet
OpenGVLab/InternVL3_5-4B-Instruct.
Text-side linears are compressed-tensors W4A16. The vision tower and multimodal connector stay BF16.1docker run --rm -it \
2 --gpus all \
3 --ipc=host \
4 -p 8001:8000 \
5 -v ~/.cache/huggingface:/root/.cache/huggingface \
6 vllm/vllm-openai:latest \
7 OpenGVLab/InternVL3_5-4B-Instruct \
8 --served-model-name InternVL3_5-4B-Instruct-stock \
9 --trust-remote-code \
10 --dtype bfloat16 \
11 --max-model-len 40961docker run --rm -it \
2 --gpus all \
3 --ipc=host \
4 -p 8001:8000 \
5 -v /path/to/InternVL3_5-4B-Instruct-W4A16-BF16Vision:/model \
6 -v ~/.cache/huggingface:/root/.cache/huggingface \
7 vllm/vllm-openai:latest \
8 --model /model \
9 --served-model-name InternVL3_5-4B-Instruct-W4A16-BF16Vision \
10 --trust-remote-code \
11 --dtype bfloat16 \
12 --quantization compressed-tensors \
13 --max-model-len 4096 \
14 --limit-mm-per-prompt '{"image":1}' \
15 --allowed-local-media-path /python verify.py --url http://localhost:8001/v1/chat/completionssmoke_image.png asset used for verification; the image test uses that file directly.