This Scruples dataset is a filtered version of metaeval/scruples which add in binary labels for classification task "Is The author in the wrong?" instead of the original "Who's in the wrong".
This dataset test split is a merge of the original validation and test split where we filtered out rows with less than 5 human labels and labels that are in a middle (neutral). We also downsample the labels so that the binary labels are evenly distributed. Here is the original code to filter the dataset:… See the full description on the dataset page:
https://huggingface.co/datasets/justinphan3110/scruples.
