You can deploy this model and start earning money today!

RichardErkhov/
mii-llm_-_maestrale-chat-v0.3-beta-gguf

Name: mii-llm_-_maestrale-chat-v0.3-beta-gguf
Availability: InStock

Views

No views yet

Quantization made by Richard Erkhov.

Github

Discord

Request more models

maestrale-chat-v0.3-beta - GGUF

Model creator: https://huggingface.co/mii-llm/
Original model: https://huggingface.co/mii-llm/maestrale-chat-v0.3-beta/

Name	Quant method	Size
maestrale-chat-v0.3-beta.Q2_K.gguf	Q2_K	2.53GB
maestrale-chat-v0.3-beta.Q3_K_S.gguf	Q3_K_S	2.95GB
maestrale-chat-v0.3-beta.Q3_K.gguf	Q3_K	3.28GB
maestrale-chat-v0.3-beta.Q3_K_M.gguf	Q3_K_M	3.28GB
maestrale-chat-v0.3-beta.Q3_K_L.gguf	Q3_K_L	3.56GB
maestrale-chat-v0.3-beta.IQ4_XS.gguf	IQ4_XS	3.67GB
maestrale-chat-v0.3-beta.Q4_0.gguf	Q4_0	3.83GB
maestrale-chat-v0.3-beta.IQ4_NL.gguf	IQ4_NL	3.87GB
maestrale-chat-v0.3-beta.Q4_K_S.gguf	Q4_K_S	3.86GB
maestrale-chat-v0.3-beta.Q4_K.gguf	Q4_K	4.07GB
maestrale-chat-v0.3-beta.Q4_K_M.gguf	Q4_K_M	4.07GB
maestrale-chat-v0.3-beta.Q4_1.gguf	Q4_1	4.24GB
maestrale-chat-v0.3-beta.Q5_0.gguf	Q5_0	4.65GB
maestrale-chat-v0.3-beta.Q5_K_S.gguf	Q5_K_S	4.65GB
maestrale-chat-v0.3-beta.Q5_K.gguf	Q5_K	4.78GB
maestrale-chat-v0.3-beta.Q5_K_M.gguf	Q5_K_M	4.78GB
maestrale-chat-v0.3-beta.Q5_1.gguf	Q5_1	5.07GB
maestrale-chat-v0.3-beta.Q6_K.gguf	Q6_K	5.53GB
maestrale-chat-v0.3-beta.Q8_0.gguf	Q8_0	7.17GB

Original model description:

language:

it license: cc-by-nc-4.0 tags:
sft
it
mistral
chatml
axolotl
kpo prompt_template: <|im_start|>system {system_message}<|im_end|> <|im_start|>user {prompt}<|im_end|> <|im_start|>assistant model-index:
name: maestrale-chat-v0.3-beta results: []

Want to contribute? Please donate! This will let us work on better datasets and models!

Maestrale chat beta ༄

By @efederici and @mferraretto

Model description

Language Model: Mistral-7b for the Italian language, continued pre-training for Italian on a curated large-scale high-quality corpus.
Fine-Tuning: SFT performed on convs/instructions for three epochs.
KTO: Aligned with KTO.

v0.3

Function calling
Reduced default system prompt to avoid wasting tokens (pre-alignment)

This model uses ChatML prompt format:

<|im_start|>system
Sei un assistente utile.<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

Usage:

python

from transformers import (
    AutoTokenizer, 
    AutoModelForCausalLM, 
    GenerationConfig,
    TextStreamer
)
import torch

tokenizer = AutoTokenizer.from_pretrained("mii-llm/maestrale-chat-v0.3-beta")
model = AutoModelForCausalLM.from_pretrained("mii-llm/maestrale-chat-v0.3-beta", load_in_8bit=True, device_map="auto")

gen = GenerationConfig(
    do_sample=True,
    temperature=0.7,
    repetition_penalty=1.2,
    top_k=50,
    top_p=0.95,
    max_new_tokens=500,
    pad_token_id=tokenizer.eos_token_id,
    eos_token_id=tokenizer.convert_tokens_to_ids("<|im_end|>")
)

streamer = TextStreamer(tokenizer, skip_prompt=True)

messages = [
    {"role": "system", "content": "Sei un assistente utile."},
    {"role": "user", "content": "{prompt}"}
]

with torch.no_grad():
    temp = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = tokenizer(temp, return_tensors="pt").to("cuda")

    _ = model.generate(
        **inputs,
        streamer=streamer,
        generation_config=gen
    )

Intended uses & limitations

It's a beta version, but it's not safe.

RichardErkhov/mii-llm_-_maestrale-chat-v0.3-beta-gguf