50K samples selected from ConicCat/TuluAmoral728K using MIG.
All intentional safety data and most if not all refusals should have been cleaned from this dataset. I also chose to exclude the Aya dataset, as I found the quality of Aya to be subpar.
The samples in this dataset are ordered, so you should pull samples BEFORE you shuffle with something like dataset = load_dataset('ConicCat/TuluAmoral50K-MIG', split='train[:10000]').
Also the MIG paper authors found that exactly three epochs was… See the full description on the dataset page:
https://huggingface.co/datasets/ConicCat/TuluAmoral50K-MIG.
