Name	Quant method	Size
prometheus-8x7b-v2.0.Q2_K.gguf	Q2_K	16.12GB
prometheus-8x7b-v2.0.IQ3_XS.gguf	IQ3_XS	18.02GB
prometheus-8x7b-v2.0.IQ3_S.gguf	IQ3_S	19.03GB
prometheus-8x7b-v2.0.Q3_K_S.gguf	Q3_K_S	19.03GB
prometheus-8x7b-v2.0.IQ3_M.gguf	IQ3_M	19.96GB
prometheus-8x7b-v2.0.Q3_K.gguf	Q3_K	21.0GB
prometheus-8x7b-v2.0.Q3_K_M.gguf	Q3_K_M	21.0GB
prometheus-8x7b-v2.0.Q3_K_L.gguf	Q3_K_L	22.51GB
prometheus-8x7b-v2.0.IQ4_XS.gguf	IQ4_XS	23.63GB
prometheus-8x7b-v2.0.Q4_0.gguf	Q4_0	24.63GB
prometheus-8x7b-v2.0.IQ4_NL.gguf	IQ4_NL	24.91GB
prometheus-8x7b-v2.0.Q4_K_S.gguf	Q4_K_S	24.91GB
prometheus-8x7b-v2.0.Q4_K.gguf	Q4_K	26.49GB
prometheus-8x7b-v2.0.Q4_K_M.gguf	Q4_K_M	26.49GB
prometheus-8x7b-v2.0.Q4_1.gguf	Q4_1	27.32GB
prometheus-8x7b-v2.0.Q5_0.gguf	Q5_0	30.02GB
prometheus-8x7b-v2.0.Q5_K_S.gguf	Q5_K_S	30.02GB
prometheus-8x7b-v2.0.Q5_K.gguf	Q5_K	30.95GB
prometheus-8x7b-v2.0.Q5_K_M.gguf	Q5_K_M	30.95GB
prometheus-8x7b-v2.0.Q5_1.gguf	Q5_1	32.71GB
prometheus-8x7b-v2.0.Q6_K.gguf	Q6_K	35.74GB
prometheus-8x7b-v2.0.Q8_0.gguf	Q8_0	46.22GB

Original model description:

tags:

text2text-generation datasets:
prometheus-eval/Feedback-Collection
prometheus-eval/Preference-Collection license: apache-2.0 language:
en pipeline_tag: text2text-generation library_name: transformers metrics:
pearsonr
spearmanr
kendall-tau
accuracy

Links for Reference

Homepage: In Progress
Repository:https://github.com/prometheus-eval/prometheus-eval
Paper:https://arxiv.org/abs/2405.01535
Point of Contact:seungone@cmu.edu

TL;DR

Prometheus 2 is an alternative of GPT-4 evaluation when doing fine-grained evaluation of an underlying LLM & a Reward model for Reinforcement Learning from Human Feedback (RLHF).

Prometheus 2 is a language model using Mistral-Instruct as a base model. It is fine-tuned on 100K feedback within the Feedback Collection and 200K feedback within the Preference Collection. It is also made by weight merging to support both absolute grading (direct assessment) and relative grading (pairwise ranking). The surprising thing is that we find weight merging also improves performance on each format.

Model Details

Model Description

Model type: Language model
Language(s) (NLP): English
License: Apache 2.0
Related Models: All Prometheus Checkpoints
Resources for more information:
- Research paper
- GitHub Repo

Prometheus is trained with two different sizes (7B and 8x7B). You could check the 7B sized LM on this page. Also, check out our dataset as well on this page and this page.

Prompt Format

We have made wrapper functions and classes to conveniently use Prometheus 2 at our github repository. We highly recommend you use it!

However, if you just want to use the model for your use case, please refer to the prompt format below. Note that absolute grading and relative grading requires different prompt templates and system prompts.

Absolute Grading (Direct Assessment)

Prometheus requires 4 components in the input: An instruction, a response to evaluate, a score rubric, and a reference answer. You could refer to the prompt format below. You should fill in the instruction, response, reference answer, criteria description, and score description for score in range of 1 to 5.

Fix the components with {text} inside.

###Task Description:
An instruction (might include an Input inside it), a response to evaluate, a reference answer that gets a score of 5, and a score rubric representing a evaluation criteria are given.
1. Write a detailed feedback that assess the quality of the response strictly based on the given score rubric, not evaluating in general.
2. After writing a feedback, write a score that is an integer between 1 and 5. You should refer to the score rubric.
3. The output format should look as follows: \"Feedback: (write a feedback for criteria) [RESULT] (an integer number between 1 and 5)\"
4. Please do not generate any other opening, closing, and explanations.

###The instruction to evaluate:
{orig_instruction}

###Response to evaluate:
{orig_response}

###Reference Answer (Score 5):
{orig_reference_answer}

###Score Rubrics:
[{orig_criteria}]
Score 1: {orig_score1_description}
Score 2: {orig_score2_description}
Score 3: {orig_score3_description}
Score 4: {orig_score4_description}
Score 5: {orig_score5_description}

###Feedback:

After this, you should apply the conversation template of Mistral (not applying it might lead to unexpected behaviors). You can find the conversation class at this link.

conv = get_conv_template("mistral")
conv.set_system_message("You are a fair judge assistant tasked with providing clear, objective feedback based on specific criteria, ensuring each assessment reflects the absolute standards set for performance.")
conv.append_message(conv.roles[0], dialogs['instruction'])
conv.append_message(conv.roles[1], None)
prompt = conv.get_prompt()

x = tokenizer(prompt,truncation=False)

As a result, a feedback and score decision will be generated, divided by a separating phrase [RESULT]

Relative Grading (Pairwise Ranking)

Prometheus requires 4 components in the input: An instruction, 2 responses to evaluate, a score rubric, and a reference answer. You could refer to the prompt format below. You should fill in the instruction, 2 responses, reference answer, and criteria description.

Fix the components with {text} inside.

###Task Description:
An instruction (might include an Input inside it), a response to evaluate, and a score rubric representing a evaluation criteria are given.
1. Write a detailed feedback that assess the quality of two responses strictly based on the given score rubric, not evaluating in general.
2. After writing a feedback, choose a better response between Response A and Response B. You should refer to the score rubric.
3. The output format should look as follows: "Feedback: (write a feedback for criteria) [RESULT] (A or B)"
4. Please do not generate any other opening, closing, and explanations.

###Instruction:
{orig_instruction}

###Response A:
{orig_response_A}

###Response B:
{orig_response_B}

###Reference Answer:
{orig_reference_answer}

###Score Rubric:
{orig_criteria}

###Feedback:

After this, you should apply the conversation template of Mistral (not applying it might lead to unexpected behaviors). You can find the conversation class at this link.

conv = get_conv_template("mistral")
conv.set_system_message("You are a fair judge assistant assigned to deliver insightful feedback that compares individual performances, highlighting how each stands relative to others within the same cohort.")
conv.append_message(conv.roles[0], dialogs['instruction'])
conv.append_message(conv.roles[1], None)
prompt = conv.get_prompt()

x = tokenizer(prompt,truncation=False)

As a result, a feedback and score decision will be generated, divided by a separating phrase [RESULT]

License

Feedback Collection, Preference Collection, and Prometheus 2 are subject to OpenAI's Terms of Use for the generated data. If you suspect any violations, please reach out to us.

Citation

If you find the following model helpful, please consider citing our paper!

BibTeX:

bibtex

@misc{kim2023prometheus,
    title={Prometheus: Inducing Fine-grained Evaluation Capability in Language Models},
    author={Seungone Kim and Jamin Shin and Yejin Cho and Joel Jang and Shayne Longpre and Hwaran Lee and Sangdoo Yun and Seongjin Shin and Sungdong Kim and James Thorne and Minjoon Seo},
    year={2023},
    eprint={2310.08491},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}