quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }
model = AutoAWQForCausalLM.from_pretrained(merged_path, **{"low_cpu_mem_usage": True, "use_cache": False}, 
attn_implementation="flash_attention_2", torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(lora_path, trust_remote_code=False)
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)

Library versions used for quantization

raw

autoawq                   0.2.6
autoawq_kernels           0.0.7

Running the AWQ model

First install the flow judge library

shell

git clone https://github.com/flowaicom/flow-judge
cd flow-judge
pip install -e ".[vllm]"

Quickstart with Python:

python

from flow_judge import Vllm, Llamafile, Hf, EvalInput, FlowJudge
from flow_judge.metrics import RESPONSE_FAITHFULNESS_5POINT
from IPython.display import Markdown, display

# If you are running on an Ampere GPU or newer, create a model using VLLM
model = Vllm(quantization=True)

# If you have other applications open taking up VRAM, you can use less VRAM by setting gpu_memory_utilization to a lower value.
# model = Vllm(gpu_memory_utilization=0.70)

# Or create a model using Llamafile if not running an Nvidia GPU & running a Silicon MacOS for example
# model = Llamafile()

# Initialize the judge
faithfulness_judge = FlowJudge(
    metric=RESPONSE_FAITHFULNESS_5POINT,
    model=model
)

# Sample to evaluate
query = ...
context = ...
response = ...

# Create an EvalInput
# We want to evaluate the response to the customer issue based on the context and the user instructions
eval_input = EvalInput(
    inputs=[
        {"query": query},
        {"context": context},
    ],
    output={"response": response},
)

# Run the evaluation
result = faithfulness_judge.evaluate(eval_input, save_results=False)

# Display the result
display(Markdown(f"__Feedback:__\n{result.feedback}\n\n__Score:__\n{result.score}"))

Discover more at our repository https://github.com/flowaicom/flow-judge

Original model card: Flow-Judge-v0.1

🚀 Flow Judge | 📄 Technical report | 💻 flow-judge

Model Summary

Flow-Judge-v0.1 is a compact yet powerful 3.8B model that offers customizable LLM system evaluations across various fields. The model inherits it's architecture from Phi-3.5-mini instruct model which enables Flow-Judge to deliver high-quality results while maintaining a small footprint. Despite its smaller size, it achieves performance comparable to larger models in both held-out and out-of-domain benchmarks. Flow-Judge-v0.1 supports multiple scoring scales, provides qualitative feedback, and generates structured evaluation outputs. Trained on a smaller synthetic dataset, it represents an efficient approach to AI development. Released under the Apache 2.0 license, Flow Judge is an open and accessible model suitable for developers and companies seeking cost-effective and rapid evaluations using custom rubrics.

Quantized weights

Quickstart

Quickstart

Intended Use Case

Flow Judge is intended to be used on custom LLM system evaluation tasks.

Customizable evaluations: Users can define their own evaluation criteria and rubrics, tailoring Flow Judge to their specific needs and requirements. This flexibility allows for the creation of highly targeted assessments that accurately measure performance of their LLM system
Flow Judge supports three different scoring scales:
- Pass/fail: Suitable for binary assessments, such as determining whether a piece of text meets a specific standard or contains errors.
- 3-Likert: Allows for more granular evaluations, with scores ranging from negative to neutral to positive. Useful for assessing the overall quality or sentiment of a piece of text.
- 5-Likert: Provides an even more nuanced assessment, with scores ranging from strongly negative to strongly positive, enabling users to capture subtle differences in quality or sentiment.
Easy to interpret results:
- Flow Judge produces structured evaluations with <feedback> and <score> tags.
  - Qualitative feedback: Flow Judge detects errors and grades outputs and provides qualitative feedback that explains its reasoning for assigning a particular score from the rubric while highlighting problematic parts of the responses.
  - Score: Based on a grading rubric Flow Judge will return a numerical score on binary, likert-3 or likert-5 scale.

Training

Model

Flow Judge is based on the Phi-3.5-mini architecture, and the base model checkpoint used is specifically its instruct version. The model uses the same tokenizer, supports MQA and Flash Attention 2, and has weights in bfloat16 precision. However, post-finetuning, the model's support for languages and long context lengths has not been fully tested. Due to specialized Supervised Fine-Tuning (SFT), Flow Judge might show different benchmark results and support a maximum context length of 8192, shorter than the base model's.

Training Datasets

Flow-Judge-v0.1 has been trained on synthetically generated datasets. The construction of training datasets for Flow Judge involves a multi-step process:

Manually curating seed rubrics to serve as a foundation
Synthetically generating domain-adapted metrics and rubrics for various domains
Synthetically generating training instances with multiple inputs, such as user queries and contextual information
Employing a dual-evaluation strategy with consensus to ensure quality and consistency

This process creates a comprehensive and diverse set of training instances that enable accurate, domain-specific evaluations of LLM systems in generative AI products while minimizing human intervention.

Read more about the dataset construction from here

Fine-tuning

For fine-tuning we used Axolotl's preprocessing to ensure input training data is consistent. We then conducted supervised fine-tuning based on microsoft/Phi-3.5-mini-instruct using RSLoRa. More detailed information about the fine-tuning process is provided in our technical report.

Usage

Prompt format

Prompt template with inputs

text

# GOAL
Your job is to evaluate a task carried out by an AI system powered by a large language model.
You will be provided with the inputs and output of the task, as well as the evaluation criteria and scoring rubric. Your task is to evaluate the output of the AI system based on the evaluation criteria and scoring rubric provided.

# INPUT
Below are the inputs required for performing the task:
<inputs>
{INPUTS}
</inputs>

# OUTPUT
Below is the output of the task:
<output>
{OUTPUT}
</output>

# EVALUATION CRITERIA AND SCORING RUBRIC
Here are the evaluation criteria and the rubric that you need to use for evaluating the task:
<evaluation_criteria>
{EVALUATION_CRITERIA}
</evaluation_criteria>

<scoring_rubric>
{RUBRIC}
</scoring_rubric>

# INSTRUCTIONS FOR THE EVALUATION
1. Understand the task and criteria: Familiarize yourself with the task to be evaluated. Review the evaluation criteria and scoring rubric to understand the different levels of performance and the descriptions for each score.
2. Review the inputs and output: Look at the inputs provided for the task. Examine the output generated from completing the task.
3. Compare output to score descriptions: Compare the output against the criteria and score descriptions in the scoring rubric. For each criterion,decide which description best matches the output.
4. After comparing the output to the score descriptions, pay attention to the small details that might impact the final score that you assign. Sometimes a small difference can dictate the final score.
5. Write verbal feedback justifying your evaluation that includes a detailed rationale, referring to specific aspects of the output and comparing them to the rubric.
6. Assign a final score based on the scoring rubric.

## FORMAT FOR THE EVALUATION
- Write the verbal feedback inside <feedback> tags without any additional surrounding text.
- Write the numeric score inside <score> tags, without any additional surrounding text and always after the feedback.

Please accurately evaluate the task. Strictly adhere to the evaluation criteria and rubric.

Prompt template without inputs

text

# GOAL
Your job is to evaluate a task carried out by an AI system powered by a large language model.

You will be provided the output of the task, as well as the evaluation criteria and scoring rubric. Your task is to evaluate the output of the AI system based on the evaluation criteria and scoring rubric provided.

# OUTPUT
Below is the output of the task:
<output>
{OUTPUT}
</output>

# EVALUATION CRITERIA AND SCORING RUBRIC
Here are the evaluation criteria and the rubric that you need to use for evaluating the task:
<evaluation_criteria>
{EVALUATION_CRITERIA}
</evaluation_criteria>

<scoring_rubric>
{RUBRIC}
</scoring_rubric>

# INSTRUCTIONS FOR THE EVALUATION
1. Understand the task and criteria: Familiarize yourself with the task to be evaluated. Review the evaluation criteria and scoring rubric to understand the different levels of performance and the descriptions for each score.
2. Review the output: Examine the output generated from completing the task.
3. Compare output to score descriptions: Compare the output against the criteria and score descriptions in the scoring rubric. For each criterion,decide which description best matches the output.
4. After comparing the output to the score descriptions, pay attention to the small details that might impact the final score that you assign. Sometimes a small difference can dictate the final score.
5. Write verbal feedback justifying your evaluation that includes a detailed rationale, referring to specific aspects of the output and comparing them to the rubric.
6. Assign a final score based on the scoring rubric.

## FORMAT FOR THE EVALUATION
- Write the verbal feedback inside <feedback> tags without any additional surrounding text.
- Write the numeric score inside <score> tags, without any additional surrounding text and always after the feedback.

Please accurately evaluate the task. Strictly adhere to the evaluation criteria and rubric.

Formatted prompt example with inputs

text

# GOAL
Your job is to evaluate a task carried out by an AI system powered by a large language model.
You will be provided with the inputs and output of the task, as well as the evaluation criteria and scoring rubric. Your task is to evaluate the output of the AI system based on the evaluation criteria and scoring rubric provided.
# INPUT/s
Below are the inputs required for performing the task:
<inputs>
<customer_query>
Subject: Issues with recent order #12345
Hello,
I recently received my order #12345, but I'm having a few problems:
1. The blue t-shirt I ordered is actually green. Can I exchange it?
2. The phone case doesn't fit my iPhone 13 Pro Max. The website said it was compatible.
3. I was charged for express shipping, but it took over a week to arrive. Can I get a refund on the shipping cost?
Please let me know how we can resolve these issues. I've been a loyal customer for years and this is the first time I've had problems like this.
Thank you,
Jane Smith
</customer_query>
</inputs>

# OUTPUT
Below is the output of the task:
<output>
Dear Jane Smith,
Thank you for reaching out about the issues with your recent order #12345. We sincerely apologize for the inconvenience you've experienced. Let me address each of your concerns:
1. T-shirt color discrepancy: We're sorry about the mix-up with the t-shirt color. We'd be happy to exchange it for the correct blue t-shirt. I'll send you a prepaid return label for the green shirt, and we'll ship out the blue one as soon as we receive the return.
2. Phone case compatibility: We apologize for the confusion regarding the phone case compatibility. Our website information should have been more accurate. We'll gladly accept a return of the incompatible case and issue a full refund for it. Additionally, I can recommend a case that will fit your iPhone 13 Pro Max if you're interested.
3. Shipping charge: You're absolutely right about the shipping charge. Since the delivery didn't meet our express shipping standards, we will refund the entire shipping cost to your original payment method. This refund should process within 3-5 business days.
To initiate these resolutions, please reply to this email confirming that you'd like to proceed with the t-shirt exchange and phone case return. Once confirmed, I'll send you the necessary return labels and process the shipping refund immediately.
We truly value your loyalty as a long-time customer and sincerely regret that you've encountered these issues. Rest assured, we're taking steps to prevent similar problems in the future. As a gesture of goodwill, we'd like to offer you a 20% discount on your next order.
If you have any further questions or concerns, please don't hesitate to reach out. We're here to ensure your complete satisfaction.
Best regards,
Alex Johnson
Customer Service Representative
</output>

# EVALUATION CRITERIA AND SCORING RUBRIC
Here are the evaluation criteria and the rubric that you need to use for evaluating the task:
<evaluation_criteria>
How well the response addresses the specific issues raised in the customer's query?
</evaluation_criteria>
<scoring_rubric>
- Score 1: The response completely fails to address the customer's needs and ignores the specific issues raised.
- Score 2: The response barely addresses the customer's query and misses most of the specific issues raised.
- Score 3: The response partially addresses the customer's query, touching on some of the specific issues but leaving others unaddressed.
- Score 4: The response adequately addresses most aspects of the customer's query and the specific issues raised.
- Score 5: The response fully and comprehensively addresses all aspects of the customer's query and all specific issues raised in a highly satisfactory manner.
</scoring_rubric>

# INSTRUCTIONS FOR THE EVALUATION
1. Understand the task and criteria: Familiarize yourself with the task to be evaluated. Review the evaluation criteria and scoring rubric to understand the different levels of performance and the descriptions for each score.
2. Review the inputs and output: Look at the inputs provided for the task. Examine the output generated from completing the task.
3. Compare output to score descriptions: Compare the output against the criteria and score descriptions in the scoring rubric. For each criterion,decide which description best matches the output.
4. After comparing the output to the score descriptions, pay attention to the small details that might impact the final score that you assign. Sometimes a small difference can dictate the final score.
5. Write verbal feedback justifying your evaluation that includes a detailed rationale, referring to specific aspects of the output and comparing them to the rubric.
6. Assign a final score based on the scoring rubric.

## FORMAT FOR THE EVALUATION
- Write the verbal feedback inside <feedback> tags without any additional surrounding text.
- Write the numeric score inside <score> tags, without any additional surrounding text and always after the feedback.
Please accurately evaluate the task. Strictly adhere to the evaluation criteria and rubric.

Note that inputs and output are formatted with XML tags. See flow-judge repository formatting functions for more details.

Inference

Evaluations can easily be run using our flow-judge library. It currently supports both Transformers and vllm engine.

To run Flow Judge efficiently, ensure your hardware meets the following requirements:

Modern GPU with at least 4 GB VRAM (e.g., NVIDIA RTX series)
Minimum of 8 GB of system memory
At least 10GB of free storage for model files and dependencies.

Evaluation

Held-out test sets

Evaluator	Pass / Fail Held-out Test set
Precision	Recall	F1
microsoft/Phi-3.5-mini-instruct	0.685	1.000	0.813
meta-llama/Meta-Llama-3.1-8B-Instruct	0.870	0.982	0.923
mistralai/Mistral-Nemo-Instruct-2407	0.709	0.994	0.827
gpt-4o-mini	0.834	1.000	0.910
flowaicom/Flow-Judge-v0.1	0.940	0.972	0.955

Evaluator	3-Likert Held-out Test set	5-Likert Held-out Test set
pearsonr	spearmanr	kendall-tau	pearsonr	spearmanr	kendall-tau
microsoft/Phi-3.5-mini-instruct	0.756	0.749	0.695	0.808	0.819	0.739
prometheus-eval/prometheus-7b-v2.0*	-	-	-	0.910	0.908	0.838
meta-llama/Meta-Llama-3.1-8B-Instruct	0.836	0.833	0.789	0.854	0.868	0.791
mistralai/Mistral-Nemo-Instruct-2407	0.813	0.807	0.758	0.870	0.867	0.789
gpt-4o-mini	0.890	0.888	0.851	0.923	0.923	0.864
flowaicom/Flow-Judge-v0.1	0.888	0.888	0.852	0.919	0.919	0.856

* Reported in model paper

RAGTruth

Evaluator	RAGTruth QA	RAGTruth Data-to-Text	RAGTruth Summarization
Precision	Recall	F1	Precision	Recall	F1	Precision	Recall	F1
microsoft/Phi-3.5-mini-instruct	0.817	0.963	0.884	0.356	1.000	0.525	0.776	1.000	0.874
meta-llama/Meta-Llama-3.1-8B-Instruct	0.844	0.986	0.910	0.382	0.537	0.447	0.797	0.940	0.863
mistralai/Mistral-Nemo-Instruct-2407	0.821	0.995	0.900	0.357	1.000	0.526	0.775	1.000	0.873
gpt-4o-mini	0.830	0.966	0.893	0.398	0.994	0.569	0.786	0.997	0.879
Luna*	0.378	0.800	0.513	0.649	0.912	0.759	0.400	0.765	0.525
RAGAS Faithfuless*	0.312	0.419	0.357	0.792	0.508	0.619	0.642	0.299	0.408
Trulens Groundedness*	0.228	0.925	0.366	0.669	0.965	0.790	0.402	0.500	0.445
flowaicom/Flow-Judge-v0.1	0.835	0.961	0.894	0.541	0.249	0.341	0.834	0.836	0.835

* reported in model paper

HaluEval, Covid-QA, PubMedQA

Evaluator	HaluEval	Covid-QA	PubMedQA
Precision	Recall	F1	Accuracy	Precision	Recall	F1	Accuracy	Precision	Recall	F1	Accuracy
microsoft/Phi-3.5-mini-instruct	0.730	0.914	0.812	0.788	0.617	0.964	0.752	0.681	0.623	0.986	0.764	0.696
meta-llama/Meta-Llama-3.1-8B-Instruct	0.864	0.891	0.878	0.874	0.663	0.976	0.790	0.734	0.681	0.962	0.797	0.750
mistralai/Mistral-Nemo-Instruct-2407	0.655	0.993	0.789	0.735	0.651	0.982	0.783	0.728	0.602	0.994	0.750	0.669
gpt-4o-mini	0.846	0.940	0.891	0.885	0.795	0.964	0.872	0.858	0.791	0.904	0.843	0.832
flowaicom/Flow-Judge-v0.1	0.826	0.895	0.859	0.854	0.767	0.877	0.818	0.807	0.874	0.624	0.728	0.767
gpt-4o*	-	-	-	0.879	-	-	-	0.821	-	-	-	0.821
Claude 3 Sonnet*	-	-	-	0.845	-	-	-	0.829	-	-	-	0.829
RAGAS Faithfulness*	-	-	-	0.706	-	-	-	0.750	-	-	-	0.669
Lynx 8B*	-	-	-	0.857	-	-	-	0.963	-	-	-	0.852
Lynx 70B*	-	-	-	0.884	-	-	-	0.975	-	-	-	0.904

* reported in model paper

Feedback Bench

Evaluator	Feedback bench
pearsonr	spearmanr	kendall-tau
microsoft/Phi-3.5-mini-instruct	0.710	0.721	0.622
prometheus-eval/prometheus-7b-v2.0*	0.878	0.909	0.773
meta-llama/Meta-Llama-3.1-8B-Instruct	0.742	0.749	0.654
mistralai/Mistral-Nemo-Instruct-2407	0.720	0.724	0.632
gpt-4o-mini	0.797	0.795	0.701
flowaicom/Flow-Judge-v0.1	0.787	0.789	0.688

* reported in model paper using reference answers

License

We opted for the Apache 2.0 license for Flow Judge to provide the community with an open, small yet powerful LM evaluator. Our goal is to support the wider adoption of rigorous evaluation techniques in LLM system development, making them more accessible to practitioners and researchers.

Limitations and future work

Multilingual evaluation: Flow Judge has been fine-tuned exclusively on English data. While the foundation model (Phi-3.5-mini-instruct [17]) may possess multilingual capabilities, we have not systematically evaluated Flow Judge performance in non-English contexts. We plan to explore multi-lingual LM evaluators in the future.

Long context and structured Inputs: Our training dataset encompasses a wide range of custom metrics relevant to evaluating LLM systems. However, it does not include examples with long context inputs or structured data formats such as JSON, since these are harder to synthetically generate. This limitation may impact Flow Judge's performance when evaluating responses that require processing extensive context or parsing structured input. Extending our model’s capabilities to handle these input types represents an important area for future research.

Math and coding: The current version has not been trained on specific task domains such as arithmetic problems or code evaluation. As a result, its performance in these specialized areas may be limited. Future iterations of the model should address these gaps.

Domain-specific knowledge and complex multi-step evaluations: Flow Judge may struggle with highly specialized domain knowledge or proprietary data outside the training scope of its foundation model. Additionally, evaluation tasks requiring multi-step reasoning or complex logical processes may challenge the model's capabilities. We strongly recommend conducting meta-evaluations of the model performance before deploying it in specialized or highly complex evaluation scenarios.

flowaicom/Flow-Judge-v0.1-AWQ

Flow-Judge-v0.1-AWQ

Description

Quantization config

Library versions used for quantization

Running the AWQ model

Original model card: Flow-Judge-v0.1

Model Summary

Intended Use Case

Training

Model

Training Datasets

Fine-tuning

Usage

Prompt format

Prompt template with inputs

Prompt template without inputs

Formatted prompt example with inputs

Inference

Evaluation

Held-out test sets

RAGTruth

HaluEval, Covid-QA, PubMedQA

Feedback Bench

License

Limitations and future work

flowaicom/
Flow-Judge-v0.1-AWQ