Here is the CUTE benchmark, a benchmark designed for testing LLM's ability to understand the characters within their tokens.
To use this dataset as we did, make sure to use tokenizer.apply_chat_template on the prompt, and then add Answer: " afterwards to the resulting string.
For smaller LLMs, you may need to do some post-processing on the final answer.
You can also check out our implementation at:
https://github.com/Leukas/cute
