Views
No views yet
1python -O -u -m vllm.entrypoints.openai.api_server \
2 --host=127.0.0.1 \
3 --port=8090 \
4 --model=Melon/Meta-Llama-3-70B-Instruct-AutoAWQ-4bit \
5 --tokenizer=meta-llama/Meta-Llama-3-70B-Instruct \
6 --tensor-parallel-size=1 \
7 --quantization awq \
8 --dtype half