optimum-rbln, ready to serve with vllm-rbln. This repo contains the compiled artifact only — you need RBLN NPUs and the RBLN SDK to run it.| NPU | 8× RBLN-CA25 (language model: 8 devices, vision tower: 8 devices, shared) |
| Compiler | rebel-compiler==0.11.0 |
| Runtime | optimum-rbln==0.11.0.post1, vllm-rbln==0.11.0, transformers==5.8.1 |
| Option | Value |
|---|---|
max_seq_len | 131,072 |
batch_size | 4 |
decoder_batch_sizes (dynamic batch) | [1, 2, 4] |
attn_impl / kvcache_partition_len | flash_attn / 16384 |
cache_impl / sliding_window | hybrid / 1024 |
prefill_chunk_size | 128 |
kvcache_num_blocks | 23 (≈377k shared KV tokens → ~2 concurrent full-131k sequences, or 4× ~94k) |
vision max_soft_tokens / image prefill | 280 / image_prefill_384 |
kvcache_num_blocks is 23, the maximum that actually loads on 8 devices: auto sizing fills every device to ~15.7 GiB with the language model alone, leaving no room for the vision tower + multimodal projector runtimes on the same devices at serve time. 23 was determined with the SDK's own per-chiplet estimator against a budget reduced by 512 MB/device reserved for the vision-side runtimes.1vllm serve <this-repo-or-local-path> \
2 --max-num-seqs 4 \
3 --max-model-len 131072 \
4 --enable-auto-tool-choice \
5 --tool-call-parser gemma4 \
6 --reasoning-parser gemma4--max-num-seqs 4 is required — vllm-rbln defaults it to 1, and the compiled decoder tops out at batch 4.--tensor-parallel-size; the 8-device split is baked into the compiled model (vllm-rbln forces TP=1).chat_template_kwargs={"enable_thinking": true}; reasoning text is returned in message.reasoning_content.