BookCorpus cleaned, near-deduplicated, and cross-deduplicated with pg19.
This dataset has also been decontaminated with respect to the following benchmarks based on n-gram overlap:
GLUE (dev set of SST-2, CoLA, QQP, WNLI, RTE, QNLI, MNLI; test set of MPRC)
SIQA, PIQA, QASC, CSQA, HellaSWAG (all dev set)
CONLL 2003
BLIMP
MAIN
BoolQ (dev set)
WinoGrande (dev set)
ANLI (test set)
ARC easy and challenge (test set)
RACE middle and high (test set)
MMLU (dev, val, and test sets)
MATH, GSM8K (test… See the full description on the dataset page:
https://huggingface.co/datasets/Geralt-Targaryen/bookcorpus.
