Llama3.1 seems to be sensitive to emoji, removing emoji from fine-tuning dataset help stabilize its behavior.
I suspect this is related to its specific tokenization schemes. (Emoji's token might be mixed up with the eos_token).
Plus, given the funny error with Llama3.1-8B-Instruct's tokenizer, it's normal that its finetuning process gets hurt like this.
