How many shots are used for evaluation of HumanEval?

[zhimin-z]

Is that 0-shot and 3-shot CoT?

[EthanC111]

Hi @zhimin-z . If you are referring to the Self-Check baselines, then yes. Please refer to https://github.com/GAIR-NLP/factool/blob/main/factool/utils/prompts/self_check.yaml

[zhimin-z]

Hi @zhimin-z . If you are referring to the Self-Check baselines, then yes. Please refer to https://github.com/GAIR-NLP/factool/blob/main/factool/utils/prompts/self_check.yaml

Thanks for your quick replies
Are these evaluation results from 0-shot or 3-shot CoT?

[EthanC111]

Hi @zhimin-z. These results were evaluated neither using 0-shot nor 3-shot. The results were evaluated using FacTool.
If you are referring to how the individual modules in FacTool are implemented, please refer to our prompts at https://github.com/GAIR-NLP/factool/tree/main/factool/utils/prompts

For some modules in certain tasks we did provide some demonstrations (e.g. query generation for KB-QA), while others we use 0-shot prompting.