There are some dubious responses in answers.json that need inspection.
To find them out, run:
import json
DUBIOUS_MARKER = "[ERROR: Failed to generate answer"
def is_dubious(answer_text: str) -> bool:
return not answer_text or DUBIOUS_MARKER in answer_text
with open("answers.json") as f
answers = json.load(f)
dubious_indices = [
idx for idx, answer in enumerate(answers) if is_dubious(answer.get("answer_text", ""))
]
These answers are mostly either null text or has this:… See the full description on the dataset page:
https://huggingface.co/datasets/sayakpaul/llm-bias-mt-bench.
