Subset of SlimPajama-6B where tokens associated with chatml prompt format are randomly added mid-text to make model forget how to do instruct and make it behave like a completion model which is not instruction following. Base Yi 1.5 models are contaminated on synthetic SFT data, hence the need for de-contamination attempts before further finetuning, if you don't want your end model to behave like ChatGPT. Should also work with Qwen 1.5 and Qwen 2 models, they are contaminated too.
