We introduce 2D-ATOMS dataset, a novel text-based dataset that evaluates machine's reasoning process under situated theory-of-mind setting.
Our dataset includes 9 different ToM evaluation tasks for each mental state under ATOMS[1], and 1 reality-checking task to test LLMs’ understanding of the world. It is important to acknowledge that our experiment serves as a proof of concept and does not aim to cover the entire spectrum of machine ToM, as our case studies are far from being exhaustive or systematic. Here we release the zero-shot version of our dataset, which is used in our paper.
