You can deploy this model and start earning money today!

RichardErkhov/
MaziyarPanahi_-_Mistral-7B-Instruct-v0.3-gguf

Name: MaziyarPanahi_-_Mistral-7B-Instruct-v0.3-gguf
Availability: InStock

Views

No views yet

Quantization made by Richard Erkhov.

Github

Discord

Request more models

Mistral-7B-Instruct-v0.3 - GGUF

Model creator: https://huggingface.co/MaziyarPanahi/
Original model: https://huggingface.co/MaziyarPanahi/Mistral-7B-Instruct-v0.3/

Name	Quant method	Size
Mistral-7B-Instruct-v0.3.Q2_K.gguf	Q2_K	2.54GB
Mistral-7B-Instruct-v0.3.IQ3_XS.gguf	IQ3_XS	2.82GB
Mistral-7B-Instruct-v0.3.IQ3_S.gguf	IQ3_S	2.97GB
Mistral-7B-Instruct-v0.3.Q3_K_S.gguf	Q3_K_S	2.95GB
Mistral-7B-Instruct-v0.3.IQ3_M.gguf	IQ3_M	3.06GB
Mistral-7B-Instruct-v0.3.Q3_K.gguf	Q3_K	3.28GB
Mistral-7B-Instruct-v0.3.Q3_K_M.gguf	Q3_K_M	3.28GB
Mistral-7B-Instruct-v0.3.Q3_K_L.gguf	Q3_K_L	3.56GB
Mistral-7B-Instruct-v0.3.IQ4_XS.gguf	IQ4_XS	3.68GB
Mistral-7B-Instruct-v0.3.Q4_0.gguf	Q4_0	3.83GB
Mistral-7B-Instruct-v0.3.IQ4_NL.gguf	IQ4_NL	3.87GB
Mistral-7B-Instruct-v0.3.Q4_K_S.gguf	Q4_K_S	3.86GB
Mistral-7B-Instruct-v0.3.Q4_K.gguf	Q4_K	4.07GB
Mistral-7B-Instruct-v0.3.Q4_K_M.gguf	Q4_K_M	4.07GB
Mistral-7B-Instruct-v0.3.Q4_1.gguf	Q4_1	4.24GB
Mistral-7B-Instruct-v0.3.Q5_0.gguf	Q5_0	4.66GB
Mistral-7B-Instruct-v0.3.Q5_K_S.gguf	Q5_K_S	4.66GB
Mistral-7B-Instruct-v0.3.Q5_K.gguf	Q5_K	4.78GB
Mistral-7B-Instruct-v0.3.Q5_K_M.gguf	Q5_K_M	4.78GB
Mistral-7B-Instruct-v0.3.Q5_1.gguf	Q5_1	5.07GB
Mistral-7B-Instruct-v0.3.Q6_K.gguf	Q6_K	5.54GB
Mistral-7B-Instruct-v0.3.Q8_0.gguf	Q8_0	7.17GB

Original model description:

license: apache-2.0 model-index:

name: Mistral-7B-Instruct-v0.3 results:
- task: type: text-generation name: Text Generation dataset: name: AI2 Reasoning Challenge (25-Shot) type: ai2_arc config: ARC-Challenge split: test args: num_few_shot: 25 metrics:
  - type: acc_norm value: 63.91 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=MaziyarPanahi/Mistral-7B-Instruct-v0.3 name: Open LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: HellaSwag (10-Shot) type: hellaswag split: validation args: num_few_shot: 10 metrics:
  - type: acc_norm value: 84.82 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=MaziyarPanahi/Mistral-7B-Instruct-v0.3 name: Open LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: MMLU (5-Shot) type: cais/mmlu config: all split: test args: num_few_shot: 5 metrics:
  - type: acc value: 62.58 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=MaziyarPanahi/Mistral-7B-Instruct-v0.3 name: Open LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: TruthfulQA (0-shot) type: truthful_qa config: multiple_choice split: validation args: num_few_shot: 0 metrics:
  - type: mc2 value: 59.45 source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=MaziyarPanahi/Mistral-7B-Instruct-v0.3 name: Open LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: Winogrande (5-shot) type: winogrande config: winogrande_xl split: validation args: num_few_shot: 5 metrics:
  - type: acc value: 78.37 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=MaziyarPanahi/Mistral-7B-Instruct-v0.3 name: Open LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: GSM8k (5-shot) type: gsm8k config: main split: test args: num_few_shot: 5 metrics:
  - type: acc value: 42.15 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=MaziyarPanahi/Mistral-7B-Instruct-v0.3 name: Open LLM Leaderboard

Model Card for Mistral-7B-Instruct-v0.3

The Mistral-7B-Instruct-v0.3 Large Language Model (LLM) is an instruct fine-tuned version of the Mistral-7B-v0.3.

Mistral-7B-v0.3 has the following changes compared to Mistral-7B-v0.2

Extended vocabulary to 32768
Supports v3 Tokenizer
Supports function calling

Installation

It is recommended to use mistralai/Mistral-7B-Instruct-v0.3 with mistral-inference. For HF transformers code snippets, please keep scrolling.

pip install mistral_inference

Download

from huggingface_hub import snapshot_download
from pathlib import Path

mistral_models_path = Path.home().joinpath('mistral_models', '7B-Instruct-v0.3')
mistral_models_path.mkdir(parents=True, exist_ok=True)

snapshot_download(repo_id="mistralai/Mistral-7B-Instruct-v0.3", allow_patterns=["params.json", "consolidated.safetensors", "tokenizer.model.v3"], local_dir=mistral_models_path)

Chat

After installing mistral_inference, a mistral-chat CLI command should be available in your environment. You can chat with the model using

mistral-chat $HOME/mistral_models/7B-Instruct-v0.3 --instruct --max_tokens 256

Instruct following

from mistral_inference.model import Transformer
from mistral_inference.generate import generate

from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
from mistral_common.protocol.instruct.messages import UserMessage
from mistral_common.protocol.instruct.request import ChatCompletionRequest


tokenizer = MistralTokenizer.from_file(f"{mistral_models_path}/tokenizer.model.v3")
model = Transformer.from_folder(mistral_models_path)

completion_request = ChatCompletionRequest(messages=[UserMessage(content="Explain Machine Learning to me in a nutshell.")])

tokens = tokenizer.encode_chat_completion(completion_request).tokens

out_tokens, _ = generate([tokens], model, max_tokens=64, temperature=0.0, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)
result = tokenizer.instruct_tokenizer.tokenizer.decode(out_tokens[0])

print(result)

Function calling

from mistral_common.protocol.instruct.tool_calls import Function, Tool
from mistral_inference.model import Transformer
from mistral_inference.generate import generate

from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
from mistral_common.protocol.instruct.messages import UserMessage
from mistral_common.protocol.instruct.request import ChatCompletionRequest


tokenizer = MistralTokenizer.from_file(f"{mistral_models_path}/tokenizer.model.v3")
model = Transformer.from_folder(mistral_models_path)

completion_request = ChatCompletionRequest(
    tools=[
        Tool(
            function=Function(
                name="get_current_weather",
                description="Get the current weather",
                parameters={
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "The city and state, e.g. San Francisco, CA",
                        },
                        "format": {
                            "type": "string",
                            "enum": ["celsius", "fahrenheit"],
                            "description": "The temperature unit to use. Infer this from the users location.",
                        },
                    },
                    "required": ["location", "format"],
                },
            )
        )
    ],
    messages=[
        UserMessage(content="What's the weather like today in Paris?"),
        ],
)

tokens = tokenizer.encode_chat_completion(completion_request).tokens

out_tokens, _ = generate([tokens], model, max_tokens=64, temperature=0.0, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)
result = tokenizer.instruct_tokenizer.tokenizer.decode(out_tokens[0])

print(result)

Generate with `transformers`

If you want to use Hugging Face transformers to generate text, you can do something like this.

from transformers import pipeline

messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you?"},
]
chatbot = pipeline("text-generation", model="mistralai/Mistral-7B-Instruct-v0.3")
chatbot(messages)

Limitations

The Mistral 7B Instruct model is a quick demonstration that the base model can be easily fine-tuned to achieve compelling performance. It does not have any moderation mechanisms. We're looking forward to engaging with the community on ways to make the model finely respect guardrails, allowing for deployment in environments requiring moderated outputs.

The Mistral AI Team

Albert Jiang, Alexandre Sablayrolles, Alexis Tacnet, Antoine Roux, Arthur Mensch, Audrey Herblin-Stoop, Baptiste Bout, Baudouin de Monicault, Blanche Savary, Bam4d, Caroline Feldman, Devendra Singh Chaplot, Diego de las Casas, Eleonore Arcelin, Emma Bou Hanna, Etienne Metzger, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Harizo Rajaona, Jean-Malo Delignon, Jia Li, Justus Murke, Louis Martin, Louis Ternon, Lucile Saulnier, Lélio Renard Lavaud, Margaret Jennings, Marie Pellat, Marie Torelli, Marie-Anne Lachaux, Nicolas Schuhl, Patrick von Platen, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Thibaut Lavril, Timothée Lacroix, Théophile Gervet, Thomas Wang, Valera Nemychnikova, William El Sayed, William Marshall

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	65.21
AI2 Reasoning Challenge (25-Shot)	63.91
HellaSwag (10-Shot)	84.82
MMLU (5-Shot)	62.58
TruthfulQA (0-shot)	59.45
Winogrande (5-shot)	78.37
GSM8k (5-shot)	42.15

RichardErkhov/MaziyarPanahi_-_Mistral-7B-Instruct-v0.3-gguf