Version: 0.3 Pairs: 420 Domains: 19 License: MIT A benchmark for measuring semantic invariance in language models. Tests whether a model gives the same answer when the same question is rephrased. This is the evaluation dataset behind the CAI Semantic Equivalence Benchmark and scored by contradish using CAI Strain v2.

What it tests

Most LLM benchmarks test accuracy. This one tests consistency. A model passes when it gives… See the full description on the dataset page: https://huggingface.co/datasets/theworkforceof/cai-semantic-equivalence-benchmark.