This dataset is the ShareGPT unfiltered dataset anon8231489123/ShareGPT_Vicuna_unfiltered, removing instances of blatant alignment and removes duplicates.
33714 instructions remain.
clean.py was first ran on hakurei/open-instruct-v1/subsets/sharegpt_data.json and then dedupe.py was ran on it.
inspired by
https://huggingface.co/datasets/ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered
All credit to anon8231489123 for the cleanup script that I adapted to wizardlm_clean.py, I then took this… See the full description on the dataset page:
https://huggingface.co/datasets/ewof/sharegpt-instruct-unfiltered-deduped.
