|
--- |
|
metrics: |
|
- accuracy |
|
--- |
|
# Hi-ToM Dataset |
|
|
|
This is the dataset for the paper "Hi-ToM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models". |
|
|
|
<img src=media/Picture1.png height=430> |
|
|
|
### The `Hi-ToM_data` folder |
|
|
|
Contains ToMh data consisting of story-question pairs and the corresponding answers. |
|
The names of subfolder branches have the following meanings: |
|
|
|
- `Tell` / `No_Tell`: whether or not the stories contain communications among agents. |
|
- `MC` / `CoT`: the prompting style. `MC` corresponds to Vanilla Prompting (VP) in the paper, while `CoT` stands for Chain-of-Thought Prompting (CoTP). |
|
- `length_n`: the story length, i.e. the number of chapters in a story. From 1 to 3. |
|
- `sample_n`: the numbering of different sample stories. |
|
- `order_n`: the ToM order of the question. From 0 to 4. |
|
|
|
### The `Hi-ToM_prompt` folder |
|
|
|
Contains prompt files that can be directly input to API. |
|
The data in it are almost the same as `Hi-ToM_data`, except that answers are eliminated. |
|
|
|
### Generate new data and prompts |
|
|
|
Run the script `generate_tomh.sh`. |
