7B parameters, 81 layers (13 hybrid attention blocks, 2 shared transformers cycling even/odd)
Hidden size: 3584, attention hidden: 7168
SSM: d_state=64, d_conv=4, ngroups=2
Requires llama.cpp with Zamba2 support (PR #21412)

Available Quantizations

Quant	Size	BPW	Prompt tok/s	Gen tok/s	Hardware
Q4_0	5.9 GB	4.53	160.2	15.2	RTX 4090
Q8_0	11 GB	8.51	147.7	14.0	RTX 4090

Sample Output (Q4_0)

Q: What is the capital of France and why is it significant?

A: The capital of France is Paris, a city that holds significant historical, cultural, and economic importance. Paris, often referred to as the "City of Light" or "City of Love," is not only the heart of France but also a global hub for art, fashion, gastronomy, and culture. Paris is renowned for its iconic landmarks such as the Eiffel Tower, the Louvre Museum, and Notre-Dame Cathedral, which attract millions of visitors each year.

Usage

bash

# Build llama.cpp with Zamba2 support
git clone https://github.com/echo313unfolding/llama.cpp -b zamba2-support
cd llama.cpp && cmake -B build -DGGML_CUDA=ON && cmake --build build -j

# Run
./build/bin/llama-cli -m zamba2-7b-instruct-v2-q4_0.gguf \
  -p "<|im_start|>user\nWhat is quantum computing?<|im_end|>\n<|im_start|>assistant\n" \
  -n 256 -ngl 999 -e --no-conversation

Conversion Details

Converted from HF safetensors using a custom Zamba2-to-GGUF converter:

Mamba-2 SSM layers: A_log to A conversion, conv1d squeeze, dt projection
2 shared transformer blocks with per-layer LoRA unfolding (W_eff = W_shared + B @ A)
Per-layer n_head_kv array (0 for Mamba layers, 32 for attention layers)
Group norm reshape for ngroups=2 (GGML column-major convention)
BPE tokenizer (v2 format)
F32 master, quantized with llama-quantize

Credits

Architecture: Zyphra (Zamba2)
GGUF Conversion: Echo Labs
llama.cpp Zamba2 support: PR #21412

EchoLabs33/Zamba2-7B-Instruct-v2-GGUF

Zamba2 7B Instruct v2 - GGUF

Architecture

Available Quantizations

Sample Output (Q4_0)

Usage

Conversion Details

Credits

EchoLabs33/
Zamba2-7B-Instruct-v2-GGUF