You can deploy this model and start earning money today!

sayyedarham/
llama3.2-1b-mamba-hybrid

Name: llama3.2-1b-mamba-hybrid
Availability: InStock

meta-llama/Llama-3.2-1B

finetune

llama3.2

Views

No views yet

llama3.2-1b-mamba-hybrid

A 16-layer hybrid of meta-llama/Llama-3.2-1B where 12 of 16 self-attention layers have been replaced with Mamba (selective state-space) blocks, then fine-tuned via knowledge distillation from the original dense model.

This is not an AutoModel-loadable checkpoint. See modeling_hybrid.py for the exact code and load instructions.

Layers kept as attention

[0, 4, 8, 12]

How to load

python

from transformers import AutoModelForCausalLM
import torch
from modeling_hybrid import hybridize

base = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-1B", torch_dtype=torch.float32
)
LAYERS_TO_KEEP_ATTENTION = set([0, 4, 8, 12])
model = hybridize(base, LAYERS_TO_KEEP_ATTENTION)
state_dict = torch.load("pytorch_model.bin", map_location="cpu")
model.load_state_dict(state_dict)
model.eval()

sayyedarham/llama3.2-1b-mamba-hybrid

llama3.2-1b-mamba-hybrid

Layers kept as attention

How to load

sayyedarham/
llama3.2-1b-mamba-hybrid