You can deploy this model and start earning money today!

RichardErkhov/
han1997_-_mamba-2.8b-zephyr-hf-gguf

Name: han1997_-_mamba-2.8b-zephyr-hf-gguf
Availability: InStock

endpoints_compatible

gguf

template

Views

No views yet

Quantization made by Richard Erkhov.

Github

Discord

Request more models

mamba-2.8b-zephyr-hf - GGUF

Model creator: https://huggingface.co/han1997/
Original model: https://huggingface.co/han1997/mamba-2.8b-zephyr-hf/

Name	Quant method	Size
mamba-2.8b-zephyr-hf.Q2_K.gguf	Q2_K	0.94GB
mamba-2.8b-zephyr-hf.Q3_K_S.gguf	Q3_K_S	1.19GB
mamba-2.8b-zephyr-hf.Q3_K.gguf	Q3_K	1.19GB
mamba-2.8b-zephyr-hf.Q3_K_M.gguf	Q3_K_M	1.19GB
mamba-2.8b-zephyr-hf.Q3_K_L.gguf	Q3_K_L	1.19GB
mamba-2.8b-zephyr-hf.IQ4_XS.gguf	IQ4_XS	1.43GB
mamba-2.8b-zephyr-hf.Q4_0.gguf	Q4_0	1.51GB
mamba-2.8b-zephyr-hf.IQ4_NL.gguf	IQ4_NL	1.51GB
mamba-2.8b-zephyr-hf.Q4_K_S.gguf	Q4_K_S	1.51GB
mamba-2.8b-zephyr-hf.Q4_K.gguf	Q4_K	1.51GB
mamba-2.8b-zephyr-hf.Q4_K_M.gguf	Q4_K_M	1.51GB
mamba-2.8b-zephyr-hf.Q4_1.gguf	Q4_1	1.66GB
mamba-2.8b-zephyr-hf.Q5_0.gguf	Q5_0	1.81GB
mamba-2.8b-zephyr-hf.Q5_K_S.gguf	Q5_K_S	1.82GB
mamba-2.8b-zephyr-hf.Q5_K.gguf	Q5_K	1.82GB
mamba-2.8b-zephyr-hf.Q5_K_M.gguf	Q5_K_M	1.82GB
mamba-2.8b-zephyr-hf.Q5_1.gguf	Q5_1	1.97GB
mamba-2.8b-zephyr-hf.Q6_K.gguf	Q6_K	2.15GB
mamba-2.8b-zephyr-hf.Q8_0.gguf	Q8_0	2.76GB

Original model description:

library_name: transformers license: apache-2.0

The following content is mostly from https://huggingface.co/state-spaces/mamba-2.8b-hf

Mamba

This repository contains the transfromers compatible mamba-2.8b-zephyr. The checkpoints are untouched, but the full config.json and tokenizer are pushed to this repo. For details of the original model before conversion, see https://huggingface.co/xiuyul/mamba-2.8b-zephyr.

Usage

You need to install transformers from main until transformers=4.39.0 is released.

pip install git+https://github.com/huggingface/transformers@main

We also recommend you to install both causal_conv_1d and mamba-ssm using:

bash

pip install causal-conv1d>=1.2.0
pip install mamba-ssm

If any of these two is not installed, the "eager" implementation will be used. Otherwise the more optimised cuda kernels will be used.

Generation

You can use the classic generate API:

python

>>> from transformers import MambaConfig, MambaForCausalLM, AutoTokenizer
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("han1997/mamba-2.8b-zephyr-hf")
>>> model = MambaForCausalLM.from_pretrained("han1997/mamba-2.8b-zephyr-hf")
>>> input_ids = tokenizer("Hey how are you doing?", return_tensors="pt")["input_ids"]

>>> out = model.generate(input_ids, max_new_tokens=10)
>>> print(tokenizer.batch_decode(out))
["Hey how are you doing?\n\nI'm doing great.\n\nI"]

PEFT finetuning example

In order to finetune using the peft library, we recommend keeping the model in float32!