Question about how to eval mPLUG-PaperOwl or other VLMs on M-paper？

[sky-fly97]

Hello, I would like to ask how to test on M-Paper dataset? For example, for the task Multimodal Diagram Analysis, its input needs to be 𝐶𝑜𝑛𝑡𝑒𝑥𝑡 + 𝐷𝑖𝑎𝑔𝑟𝑎𝑚 𝑠 + 𝑂𝑢𝑡𝑙𝑖𝑛𝑒, and the question instructions, so how are you organizing the input format for the model? Are there any associated script evaluation files about M-Paper?

[HAWLYQ]

Hi @sky-fly97 , the inputs of Multimodal Diagram Analysis are concatenated as the input of the model in the order of Context, Outline, Diagram, and Instruction. You can find samples in the file ./sft/analysis_test.jsonl. We are planning to organize and open-source the evaluation code of M-Paper recently, which may be released at the end of this month~