This repository contains validation datasets for use with the perplexity tool from the llama.cpp project.
Note: PR #5047 is required to be able to use these datasets.
The simple program in demo.cpp shows how to read these files and can be used to combine two files into one.
The simple program in convert.cpp shows how to convert the data to JSON. For instance:
g++ -o convert convert.cpp
./convert arc-easy-validation.bin arc-easy-validation.json
