List of models that have been used to calculate this set: We calculated the intersection of FSMs that all models can solve the first transition. (Judged by Qwen)
Model Name
Accuracy
claude-3-opus-20240229
94.03
gpt-4-turbo-2024-04-09
92.76
gpt-4-0125-preview
91.67
claude-3-sonnet-20240229
76.3
claude-3-haiku-20240307
69.59
CodeLlama-70b-Instruct-hf
68.12
dbrx-instruct
66.31
Qwen1.5-72B-Chat60.84
gemma-7b-it
60.66
StripedHyena-Nous-7B
60.18… See the full description on the dataset page:
https://huggingface.co/datasets/FSMBench/fsmbench_intersection_set_2.
