tdh111/

bitnet-b1.58-2B-4T-GGUF

The IQ2_BN and IQ2_BN_R4 version of microsoft/bitnet-b1.58-2B-4T-gguf for use with ik_llama.cpp.

I recommend the IQ2_BN_R4 version but you use -rtr on IQ2_BN to convert on runtime.

The chat template in the model looks incorrect (I did not change it, this is from the original Microsoft GGUF).

An example of correct usage from their transformers PR:

I was able to follow the example above and it worked for multi-turn conversations.

With the more general template (sourced from the paper) being:

<|begin_of_text|>System: {system_message}<|eot_id|> User: {user_message_1}<|eot_id|> Assistant: {assistant_message_1}<|eot_id|> User: {user_message_2}<|eot_id|> Assistant: {assistant_message_2}<|eot_id|>