Views
No views yet
A fast and lightweight Python package for evaluating question-answering models and prompting of black-box and open-source large language models.
pip install qa-metricsis all you need!
pip install qa-metrics| Method | Best For | Cost | Correlation with Human Judgment |
|---|---|---|---|
| Normalized Exact Match | Short-form QA (NQ-OPEN, HotpotQA, etc.) | Free | Good |
| PEDANTS | Both short & medium-form QA | Free | Very High |
| Neural Evaluation | Both short & long-form QA | Free | High |
| Open Source LLM Evaluation | All QA types | Free | High |
| Black-box LLM Evaluation | All QA types | Paid | Highest |
em_matchreference_answer (list of str): A list of gold (correct) answers to the questioncandidate_answer (str): The answer provided by a candidate that needs to be evaluatedboolean: True if there are any exact normalized matches between gold and candidate answers1from qa_metrics.em import em_match
2
3reference_answer = ["The Frog Prince", "The Princess and the Frog"]
4candidate_answer = "The movie \"The Princess and the Frog\" is loosely based off the Brother Grimm's \"Iron Henry\""
5match_result = em_match(reference_answer, candidate_answer)f1_score_with_precision_recallreference_answer (str): A gold (correct) answer to the questioncandidate_answer (str): The answer provided by a candidate that needs to be evaluateddictionary: Contains the F1 score, precision, and recall between a gold and candidate answerf1_matchreference_answer (list of str): List of gold answerscandidate_answer (str): Candidate answer to evaluatethreshold (float): F1 score threshold for considering a match (default: 0.5)boolean: True if F1 score exceeds threshold for any gold answer1from qa_metrics.f1 import f1_match, f1_score_with_precision_recall
2
3f1_stats = f1_score_with_precision_recall(reference_answer[0], candidate_answer)
4match_result = f1_match(reference_answer, candidate_answer, threshold=0.5)get_scorereference_answer (str): A Gold answercandidate_answer (str): Candidate answer to evaluatequestion (str): The question being evaluatedfloat: The similarity score between two strings (0 to 1)get_highest_scorereference_answer (list of str): List of gold answerscandidate_answer (str): Candidate answer to evaluatequestion (str): The question being evaluateddictionary: Contains the gold answer and candidate answer pair with highest matching scoreget_scoresreference_answer (list of str): List of gold answerscandidate_answer (str): Candidate answer to evaluatequestion (str): The question being evaluateddictionary: Contains matching scores for all gold answer and candidate answer pairsevaluatereference_answer (list of str): List of gold answerscandidate_answer (str): Candidate answer to evaluatequestion (str): The question being evaluatedboolean: True if candidate answer matches any gold answerget_question_typereference_answer (list of str): List of gold answersquestion (str): The question being evaluatedlist: The type of the question (what, who, when, how, why, which, where)get_judgement_typereference_answer (list of str): List of gold answerscandidate_answer (str): Candidate answer to evaluatequestion (str): The question being evaluatedlist: A list revised rules applicable to judge answer correctness1from qa_metrics.pedant import PEDANT
2
3pedant = PEDANT()
4scores = pedant.get_scores(reference_answer, candidate_answer, question)
5match_result = pedant.evaluate(reference_answer, candidate_answer, question)get_scorereference_answer (str): A Gold answercandidate_answer (str): Candidate answer to evaluatequestion (str): The question being evaluatedfloat: The similarity score between two strings (0 to 1)get_highest_scorereference_answer (list of str): List of gold answerscandidate_answer (str): Candidate answer to evaluatequestion (str): The question being evaluateddictionary: Contains the gold answer and candidate answer pair with highest matching scoreget_scoresreference_answer (list of str): List of gold answerscandidate_answer (str): Candidate answer to evaluatequestion (str): The question being evaluateddictionary: Contains matching scores for all gold answer and candidate answer pairstransformer_matchreference_answer (list of str): List of gold answerscandidate_answer (str): Candidate answer to evaluatequestion (str): The question being evaluatedboolean: True if transformer model considers candidate answer equivalent to any gold answer1from qa_metrics.transformerMatcher import TransformerMatcher
2
3### supports `zli12321/roberta-large-qa-evaluator`, `zli12321/answer_equivalence_bert`, `zli12321/answer_equivalence_distilbert`, `zli12321/answer_equivalence_roberta`, `zli12321/answer_equivalence_distilroberta`
4tm = TransformerMatcher("zli12321/answer_equivalence_tiny_bert")
5match_result = tm.transformer_match(reference_answer, candidate_answer, question)prompt_gptprompt (str): The input prompt textmodel_engine (str): OpenAI model to use (e.g., 'gpt-3.5-turbo')temperature (float): Controls randomness (0-1)max_tokens (int): Maximum tokens in response1from qa_metrics.prompt_llm import CloseLLM
2
3model = CloseLLM()
4model.set_openai_api_key(YOUR_OPENAI_KEY)
5result = model.prompt_gpt(prompt=prompt, model_engine='gpt-3.5-turbo')prompt_claudeprompt (str): The input prompt textmodel_engine (str): Claude model to useanthropic_version (str): API versionmax_tokens_to_sample (int): Maximum tokens in responsetemperature (float): Controls randomness (0-1)1model = CloseLLM()
2model.set_anthropic_api_key(YOUR_ANTHROPIC_KEY)
3result = model.prompt_claude(prompt=prompt, model_engine='claude-v1')promptmessage (str): The input message textmodel_engine (str): Model to usetemperature (float): Controls randomness (0-1)max_tokens (int): Maximum tokens in response1from qa_metrics.prompt_open_llm import OpenLLM
2
3model = OpenLLM()
4model.set_deepinfra_key(YOUR_DEEPINFRA_KEY)
5result = model.prompt(message=prompt, model_engine='mistralai/Mixtral-8x7B-Instruct-v0.1')1@inproceedings{li-etal-2024-pedants,
2 title = "{PEDANTS}: Cheap but Effective and Interpretable Answer Equivalence",
3 author = "Li, Zongxia and
4 Mondal, Ishani and
5 Nghiem, Huy and
6 Liang, Yijun and
7 Boyd-Graber, Jordan Lee",
8 editor = "Al-Onaizan, Yaser and
9 Bansal, Mohit and
10 Chen, Yun-Nung",
11 booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
12 month = nov,
13 year = "2024",
14 address = "Miami, Florida, USA",
15 publisher = "Association for Computational Linguistics",
16 url = "https://aclanthology.org/2024.findings-emnlp.548/",
17 doi = "10.18653/v1/2024.findings-emnlp.548",
18 pages = "9373--9398",
19 abstract = "Question answering (QA) can only make progress if we know if an answer is correct, but current answer correctness (AC) metrics struggle with verbose, free-form answers from large language models (LLMs). There are two challenges with current short-form QA evaluations: a lack of diverse styles of evaluation data and an over-reliance on expensive and slow LLMs. LLM-based scorers correlate better with humans, but this expensive task has only been tested on limited QA datasets. We rectify these issues by providing rubrics and datasets for evaluating machine QA adopted from the Trivia community. We also propose an efficient, and interpretable QA evaluation that is more stable than an exact match and neural methods (BERTScore)."
20}