Views
No views yet
microsoft/Phi-4-mini-instruct.
Text-side linears are compressed-tensors W4A16.1docker run --rm -it \
2 --gpus all \
3 --ipc=host \
4 -p 8001:8000 \
5 -v ~/.cache/huggingface:/root/.cache/huggingface \
6 vllm/vllm-openai:latest \
7 microsoft/Phi-4-mini-instruct \
8 --served-model-name Phi-4-mini-instruct-stock \
9 --dtype bfloat16 \
10 --max-model-len 4096 \
11 --gpu-memory-utilization 0.71docker run --rm -it \
2 --gpus all \
3 --ipc=host \
4 -p 8002:8000 \
5 -v /path/to/Phi-4-mini-instruct-W4A16:/model \
6 -v ~/.cache/huggingface:/root/.cache/huggingface \
7 vllm/vllm-openai:latest \
8 /model \
9 --served-model-name Phi-4-mini-instruct-W4A16 \
10 --dtype bfloat16 \
11 --quantization compressed-tensors \
12 --max-model-len 4096 \
13 --gpu-memory-utilization 0.7python verify.py --url http://localhost:8002/v1/completions