A few Q about the dataset architecture of this repository

[141forever]

Hi, congrats for your excellent work! It helps me a lot!
However, i have a few Q about the dataset architecture of this repository:

1. is ChatProtect/test/test the MainTestSet in the paper?
2. for example in path ChatProtect/test/test/sentences/chatgpt
   /Please_tell_me_about_199091_Austrian_Hockey_League_season/
   what is the meaning of m0,m1,m2 and m3?
   3.in the same path, the file 0_9_2.txt eg. what is the meaning of these three numbers?
   4.at last, could you give a specific explanation of the keys in the txt files, like "tag","wrong" and "orig_tag"?

[nielstron]

Hi, thanks for your comment.

1. Yes test/test is the MainTestSet in the paper
2. m0 - m3 refer to the different baselines for creating an alternative sentence for a sentence pair described in Appendix B in the paper "Ablation Study on Trigger". m0 - i) Continue, m1 - ii) Rephrase, m2 - iii) Q&A and m3 is our method
3. The numbering for sentence files is <num_description>_<num_sentence>_<num_generated_alternative> (all indices starting at 0). For example 0_9_2 is the
   • third (2) alternative sentence for the
   • tenth (9) sentence of the
   • first (0) description
4. The meaning is as follows:
   • tag: The ground-truth label for whether the two sentences are considered self-contradictory according to the annotation procedure described in the paper. If no ground truth was annotated (i.e. in test_big) this is the annotation generated by the aLM.
   • orig_tag: (Optional) Stores the annotation for the sentence pair as generated by the aLM specified during generation. It can be ignored as it is usually not used / overwritten by the annotation in output/test.
   • wrong: (Optional) Which of the two sentences was considered factually incorrect by annotators, either none, 1, 2 or both. Note that if the first sentence (which may occur in multiple files) is considered incorrect in any file, it overrides the annotation in other files. For example if 0_9_0 contains wrong = 1 and 0_9_1 and 0_9_2 both contain wrong = 2 then the first sentence is considered factually incorrect in 0_9_1 and 0_9_2 as well, being thus equivalent to writing wrong = both in the two files.

I hope that this information helps at understanding the architecture of the repository and please feel free to reach out if any questions remain.