Use the script generate_valid_questions.py to create an instruction set for valid questions.
python generate_valid_questions.py chemistry-by-chapter.txt valid_examples.json
Use the script generate_invalid_questions.py to create an instruction set for invalid questions.
python generate_invalid_questions.py politics.txt invalid_examples.json
Combine the two datasets.
echo -n "[" > finetune.json
cat valid_examples.json >> finetune.json
sed '$s/,$//' invalid_examples.json | cat >> finetune.json… See the full description on the dataset page:
https://huggingface.co/datasets/juntaoyuan/validate_chemistry_question.
