Zamba2-7B-Instruct-v2 is obtained from Zamba2-7B by fine-tuning on instruction-following and chat datasets.

Zamba2-7B-Instruct-v2 is a hybrid model composed of state-space (Mamba2) and transformer blocks.

The context window can be extended from 4k to 16k long-context by adjusting the rope frequency in the attention blocks (as described below).

To use Zamba2-7B-Instruct-v2, install transformers:

To install dependencies necessary to run Mamba2 kernels, install mamba-ssm from source (due to compatibility issues with PyTorch) as well as causal-conv1d:

You can run the model without using the optimized Mamba2 kernels, but it is not recommended as it will result in significantly higher latency and memory usage.

To use the context-extended version of Zamba, please load the model with use_long_context=True, i.e.:

Zamba2-7B-Instruct-v2 punches above its weight, achieving extremely strong instruction-following benchmark scores.

Moreover, due to its unique hybrid SSM architecture, Zamba2-7B-Instruct-v2 achieves extremely low inference latency and rapid generation with a significantly smaller memory footprint than comparable transformer-based models.

And memory overhead

Zamba2-7B-Instruct-v2's high performance, strong instruction-following and reasoning capabilities for its size makes it an ideal generalist small model for a wide range of applications.

Zamba2-7B-Instruct-v2 utilizes and extends our original Zamba hybrid SSM-attention architecture. The core Zamba architecture consists of a backbone of Mamba2 layers interleaved with one or more shared attention layers. This attention has shared weights to minimize the parameter cost of the model. We find that concatenating the original model embeddings to the input to this attention block improves performance, likely due to better maintenance of information across depth. The Zamba2 architecture also applies LoRA projection matrices to the shared MLP to gain some additional expressivity in each block and allow each shared block to specialize slightly to its own unique position while keeping the additional parameter overhead small.

Zamba2-7B-Instruct-v2 was trained on a mix of publicly available dataset including instruction-following and chat data. We experimented with various training approaches and found that the best recipe was as follows: