Views
No views yet
cohere2_moe: 32 layers, 128 experts, 8 active, 4 shared experts, sigmoid routing).command-a-plus-05-2026 is a vision-language model (Cohere2VisionForConditionalGeneration = a SigLIP vision encoder wrapping the Cohere2 MoE text backbone). llama.cpp's converter routes the text backbone via the text_config and produces a text-only GGUF; the vision tower is not included. If you need image input, this is not the right artifact.CohereLabs/command-a-plus-05-2026-bf16), not the W4A4 release.| File | Quant | Size | BPW | Notes |
|---|---|---|---|---|
command-a-plus-Q6_K.gguf | Q6_K | ~167 GiB | 6.56 | High-fidelity reference. No imatrix (barely moves the needle at Q6). |
command-a-plus-Q4_K_XL.gguf | Q4_K_M + pins | ~128 GiB | 5.05 | Daily driver. imatrix + per-tensor bumps (see below). |
command-a-plus-Q3_K_XL.gguf | Q3_K_M + pins | ~108 GiB | 3.97 | Fits a 128 GB box (e.g. 4× 32 GB). imatrix + q6_K pins (see below). |
*-00001-of-0000N.gguf); point llama.cpp at the first shard and it loads the rest automatically.Q4_K_M with the format-critical, always-active tensors pinned to q8_0, while the bulk routed-expert gate/up/down weights stay Q4_K (that's where the size lives):token_embd (tied output) → q8_0attn_{q,k,v,output} → q8_0ffn_gate_inp → q8_0ffn_{gate,up,down}_shexp → q8_0ffn_down_exps to q8_0 — that would push the file to ~156 GiB (near Q6_K) and defeat the daily-driver purpose.Q3_K_M with the same always-active tensors pinned to q6_K (near-lossless and ~3 GB cheaper than q8_0 at this size), routed experts at the Q3_K_M default:token_embd → q6_Kattn_{q,k,v,output} → q6_Kffn_{gate,up,down}_shexp → q6_Kffn_gate_inp → left f32 (llama.cpp's default — it's tiny and critical for MoE expert selection; don't downgrade it)llama-bench on 8× AMD Instinct MI100 (gfx908, 256 GB total VRAM), ROCm 7.1, flash-attention on, fully resident:| Quant | pp512 / pp2048 (t/s) | decode @0 / @4k / @16k ctx (t/s) |
|---|---|---|
| Q6_K | 280 / 299 | 29.1 / 27.9 / 27.4 |
| Q4_K_XL | 387 / 410 | 33.3 / 31.6 / 31.0 |
cudaMemcpyPeerAsync). If you hit a crash loading across many AMD GPUs, build llama.cpp with peer copies disabled (routes cross-device transfers through host RAM — negligible cost at decode with -sm layer):1HIPCXX="$(hipconfig -l)/clang" cmake -B build -DGGML_HIP=ON \
2 -DAMDGPU_TARGETS=gfx908 -DCMAKE_BUILD_TYPE=Release \
3 -DGGML_CUDA_NO_PEER_COPY=ON
4cmake --build build -jiommu=pt on the kernel cmdline is good practice for multi-GPU DMA but did not fix the cross-hive case on its own.)1HIPCXX="$(hipconfig -l)/clang" cmake -B build -DGGML_HIP=ON \
2 -DAMDGPU_TARGETS=gfx1030 -DCMAKE_BUILD_TYPE=Release \
3 -DGGML_CUDA_NO_PEER_COPY=ON
4cmake --build build -j-fa on works on RDNA2.-DGGML_HIP_ROCWMMA_FATTN=ON, needs librocwmma-dev): small win on CDNA (~+1.3% decode, no prefill cost). Untested on RDNA2.-DGGML_CUDA_FORCE_MMQ=ON: avoid — regressed prompt-eval ~13% on the big MoE matmuls here, no decode benefit.q8_0): not worth it — ~6% slower decode and KV is already tiny on this model; only use under genuine memory pressure.<|START_THINKING|>…<|END_THINKING|><|START_ACTION|>[{"tool_call_id":…,"tool_name":…,"parameters":…}]<|END_ACTION|>),
but current llama.cpp has no Cohere2 parser, so /v1/chat/completions returns HTTP 500 ("Failed to parse input at pos 0") instead of OpenAI tool_calls.cohere2-chat-handler.patch to common/chat.cpp and rebuild:1cd llama.cpp
2git apply /path/to/cohere2-chat-handler.patch
3cmake --build build -j --target llama-servercohere2 chat handler (detected from the template) that maps the native action format onto OpenAI tool_calls via standard_json_tools, extracts the thinking block into reasoning_content, and handles the tool-result → final-response turn. Validated for single tool calls, parallel tool calls, and the full agent loop.The includedchat_template.jinjais the model's own template (already embedded in the GGUF); it's provided for reference. Tool-call parsing is done by the C++ handler above, not the template.
1./build/bin/llama-server -m command-a-plus-Q4_K_XL-00001-of-0000N.gguf \
2 -ngl 999 -fa on -c 16384 --jinja --reasoning-format deepseek \
3 --host 0.0.0.0 --port 8080--jinja applies the Cohere chat template (special tokens, citations, tool formatting).--reasoning-format deepseek surfaces the <|START_THINKING|> block as reasoning_content.text_config to the Cohere2MoeForCausalLM handler).TokenizersBackend tokenizer class.