Views
No views yet
| Parameter | Value |
|---|---|
| Model | Qwen/Qwen3-8B |
| Tensor Parallel | 2 |
| Max Model Length | 4096 |
| Max Num Seqs | 8 |
| Block Size | 32 |
| Data Type | bf16 (auto-cast matmul) |
| Compiler Opt | -O1 |
1python -m vllm.entrypoints.openai.api_server \
2 --model /path/to/this/repo \
3 --max-model-len 4096 \
4 --tensor-parallel-size 2 \
5 --max-num-seqs 8 \
6 --block-size 32 \
7 --num-gpu-blocks-override 8 \
8 --no-enable-prefix-cachingNEURON_COMPILED_ARTIFACTS to neuron-compiled-artifacts/ to skip recompilation.config.json, tokenizer.json, ... # Model config & tokenizer
model.safetensors # Dummy (satisfies transformers validation)
neuron-compiled-artifacts/
neuron_config.json # Neuron compilation config
model.pt # Compiled NEFF (~159 MB)
weights/
tp0_sharded_checkpoint.safetensors # Shard 0 (~8 GB)
tp1_sharded_checkpoint.safetensors # Shard 1 (~8 GB)