This dataset is Anthropic/hh-rlhf unfiltered and deduped, removing 18170 instances of blatant alignment and 51954 duplicates.
99228 instructions remain.
i downloaded all the data (except red team) from
https://huggingface.co/datasets/Anthropic/hh-rlhf/tree/09be8c5bbc57cb3887f3a9732ad6aa7ec602a1fa unarchived it merged it all into one file then ran clean.py and then ran dedupe.py to get the resulting file
inspired by… See the full description on the dataset page:
https://huggingface.co/datasets/ewof/hh-rlhf-instruct-unfiltered-deduped.
