By providing proper credit and attribution, you are granted permission to use this model as a foundational base for further DPO/PPO tuning or Merges. I actively encourage users to customize and enhance the model according to their specific needs, as this version is designed to be a comprehensive, fully fine-tuned general model. Dive in and innovate!

Evaluation

We evaluated this model on a wide range of tasks using Language Model Evaluation Harness from EleutherAI.

Here are the results on similar metrics used by HuggingFaceH4 Open LLM Leaderboard

Metric	Value
Avg.	67.28
AI2 Reasoning Challenge (25-Shot)	60.92
HellaSwag (10-Shot)	81.78
MMLU (5-Shot)	64.97
TruthfulQA (0-shot)	55.04
Winogrande (5-shot)	73.40
GSM8k (5-shot)	67.55

Example Usage

Here is the ChatML prompt format

<|im_start|>system
You are Orca Mini, a helpful AI assistant.<|im_end|>
<|im_start|>user
Hello Orca Mini, what can you do for me?<|im_end|>
<|im_start|>assistant

Below shows a code example on how to use this model

python

from transformers import AutoModel, AutoTokenizer
model_slug = "pankajmathur/orca_mini_v5_8b"
model = AutoModel.from_pretrained(model_slug)
tokenizer = AutoTokenizer.from_pretrained(model_slug)

messages = [
    {"role": "system", "content": "You are Orca Mini, a helpful AI assistant."},
    {"role": "user", "content": "Hello Orca Mini, what can you do for me?"}
]

gen_input = tokenizer.apply_chat_template(messages, return_tensors="pt")
model.generate(**gen_input)

This model is governed by META LLAMA 3 COMMUNITY LICENSE AGREEMENT

Quants

GGUF : Coming Soon

AWQ: Coming Soon

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	20.16
IFEval (0-Shot)	48.06
BBH (3-Shot)	29.35
MATH Lvl 5 (4-Shot)	7.85
GPQA (0-shot)	4.92
MuSR (0-shot)	7.70
MMLU-PRO (5-shot)	23.07