result confusion?

[tiandamiao]

After executing the original code, there is a difference between the results in the paper and the paper. what should I do?

[BIGBALLON]

this is my training results:

airplane, 100, 1, 50, 1

   Avg Run Time (ms/batch): 4.985 AUC: 0.975 max AUC: 0.975
   Avg Run Time (ms/batch): 4.649 AUC: 0.989 max AUC: 0.989
   Avg Run Time (ms/batch): 4.377 AUC: 0.909 max AUC: 0.989
   Avg Run Time (ms/batch): 4.689 AUC: 0.843 max AUC: 0.989
   Avg Run Time (ms/batch): 4.659 AUC: 0.784 max AUC: 0.989
   Avg Run Time (ms/batch): 4.813 AUC: 0.751 max AUC: 0.989
   Avg Run Time (ms/batch): 4.538 AUC: 0.769 max AUC: 0.989
   Avg Run Time (ms/batch): 4.602 AUC: 0.732 max AUC: 0.989
   Avg Run Time (ms/batch): 5.159 AUC: 0.729 max AUC: 0.989
   Avg Run Time (ms/batch): 4.722 AUC: 0.792 max AUC: 0.989
   Avg Run Time (ms/batch): 4.528 AUC: 0.733 max AUC: 0.989
   Avg Run Time (ms/batch): 4.634 AUC: 0.774 max AUC: 0.989
   Avg Run Time (ms/batch): 4.834 AUC: 0.747 max AUC: 0.989
   Avg Run Time (ms/batch): 4.509 AUC: 0.764 max AUC: 0.989
   Avg Run Time (ms/batch): 4.762 AUC: 0.736 max AUC: 0.989
   Avg Run Time (ms/batch): 4.465 AUC: 0.748 max AUC: 0.989

another run:

Running CIFAR. Anomaly Class: airplane 
Files already downloaded and verified
Files already downloaded and verified
>> Training skip-ganomaly on cifar10 to detect airplane
>> Training model skip-ganomaly. Epoch 1/25                                                                                                                                                                          
   Testing skip-ganomaly
   Avg Run Time (ms/batch): 22.189 AUC: 0.992 max AUC: 0.992
>> Training model skip-ganomaly. Epoch 2/25                                                                                                                                                                          
   Testing skip-ganomaly
   Avg Run Time (ms/batch): 22.255 AUC: 0.961 max AUC: 0.992
>> Training model skip-ganomaly. Epoch 3/25                                                                                                                                                                          
   Testing skip-ganomaly
   Avg Run Time (ms/batch): 22.382 AUC: 0.886 max AUC: 0.992
>> Training model skip-ganomaly. Epoch 4/25                                                                                                                                                                          
   Testing skip-ganomaly
   Avg Run Time (ms/batch): 22.284 AUC: 0.827 max AUC: 0.992

is that right? the AUC is not increased?? and the AUC after the first epoch is 0.975?

[BIGBALLON]

@tiandamiao @samet-akcay is there anything wrong? can you give me some help?

[davids-zhou]

i have the same problem. the first epoch is a high auc. then auc is decreased. why?

[BIGBALLON]

@davids-zhou and I shuffle the test data, then the result is low.

[JJY0710]

这是我的训练结果：

airplane, 100, 1, 50, 1

   Avg Run Time (ms/batch): 4.985 AUC: 0.975 max AUC: 0.975
   Avg Run Time (ms/batch): 4.649 AUC: 0.989 max AUC: 0.989
   Avg Run Time (ms/batch): 4.377 AUC: 0.909 max AUC: 0.989
   Avg Run Time (ms/batch): 4.689 AUC: 0.843 max AUC: 0.989
   Avg Run Time (ms/batch): 4.659 AUC: 0.784 max AUC: 0.989
   Avg Run Time (ms/batch): 4.813 AUC: 0.751 max AUC: 0.989
   Avg Run Time (ms/batch): 4.538 AUC: 0.769 max AUC: 0.989
   Avg Run Time (ms/batch): 4.602 AUC: 0.732 max AUC: 0.989
   Avg Run Time (ms/batch): 5.159 AUC: 0.729 max AUC: 0.989
   Avg Run Time (ms/batch): 4.722 AUC: 0.792 max AUC: 0.989
   Avg Run Time (ms/batch): 4.528 AUC: 0.733 max AUC: 0.989
   Avg Run Time (ms/batch): 4.634 AUC: 0.774 max AUC: 0.989
   Avg Run Time (ms/batch): 4.834 AUC: 0.747 max AUC: 0.989
   Avg Run Time (ms/batch): 4.509 AUC: 0.764 max AUC: 0.989
   Avg Run Time (ms/batch): 4.762 AUC: 0.736 max AUC: 0.989
   Avg Run Time (ms/batch): 4.465 AUC: 0.748 max AUC: 0.989

另一轮：

Running CIFAR. Anomaly Class: airplane 
Files already downloaded and verified
Files already downloaded and verified
>> Training skip-ganomaly on cifar10 to detect airplane
>> Training model skip-ganomaly. Epoch 1/25                                                                                                                                                                          
   Testing skip-ganomaly
   Avg Run Time (ms/batch): 22.189 AUC: 0.992 max AUC: 0.992
>> Training model skip-ganomaly. Epoch 2/25                                                                                                                                                                          
   Testing skip-ganomaly
   Avg Run Time (ms/batch): 22.255 AUC: 0.961 max AUC: 0.992
>> Training model skip-ganomaly. Epoch 3/25                                                                                                                                                                          
   Testing skip-ganomaly
   Avg Run Time (ms/batch): 22.382 AUC: 0.886 max AUC: 0.992
>> Training model skip-ganomaly. Epoch 4/25                                                                                                                                                                          
   Testing skip-ganomaly
   Avg Run Time (ms/batch): 22.284 AUC: 0.827 max AUC: 0.992

那正确吗？没有增加AUC ？？？而第一个时期之后的AUC是0.975？

这是我的训练结果：

airplane, 100, 1, 50, 1

   Avg Run Time (ms/batch): 4.985 AUC: 0.975 max AUC: 0.975
   Avg Run Time (ms/batch): 4.649 AUC: 0.989 max AUC: 0.989
   Avg Run Time (ms/batch): 4.377 AUC: 0.909 max AUC: 0.989
   Avg Run Time (ms/batch): 4.689 AUC: 0.843 max AUC: 0.989
   Avg Run Time (ms/batch): 4.659 AUC: 0.784 max AUC: 0.989
   Avg Run Time (ms/batch): 4.813 AUC: 0.751 max AUC: 0.989
   Avg Run Time (ms/batch): 4.538 AUC: 0.769 max AUC: 0.989
   Avg Run Time (ms/batch): 4.602 AUC: 0.732 max AUC: 0.989
   Avg Run Time (ms/batch): 5.159 AUC: 0.729 max AUC: 0.989
   Avg Run Time (ms/batch): 4.722 AUC: 0.792 max AUC: 0.989
   Avg Run Time (ms/batch): 4.528 AUC: 0.733 max AUC: 0.989
   Avg Run Time (ms/batch): 4.634 AUC: 0.774 max AUC: 0.989
   Avg Run Time (ms/batch): 4.834 AUC: 0.747 max AUC: 0.989
   Avg Run Time (ms/batch): 4.509 AUC: 0.764 max AUC: 0.989
   Avg Run Time (ms/batch): 4.762 AUC: 0.736 max AUC: 0.989
   Avg Run Time (ms/batch): 4.465 AUC: 0.748 max AUC: 0.989

另一轮：

Running CIFAR. Anomaly Class: airplane 
Files already downloaded and verified
Files already downloaded and verified
>> Training skip-ganomaly on cifar10 to detect airplane
>> Training model skip-ganomaly. Epoch 1/25                                                                                                                                                                          
   Testing skip-ganomaly
   Avg Run Time (ms/batch): 22.189 AUC: 0.992 max AUC: 0.992
>> Training model skip-ganomaly. Epoch 2/25                                                                                                                                                                          
   Testing skip-ganomaly
   Avg Run Time (ms/batch): 22.255 AUC: 0.961 max AUC: 0.992
>> Training model skip-ganomaly. Epoch 3/25                                                                                                                                                                          
   Testing skip-ganomaly
   Avg Run Time (ms/batch): 22.382 AUC: 0.886 max AUC: 0.992
>> Training model skip-ganomaly. Epoch 4/25                                                                                                                                                                          
   Testing skip-ganomaly
   Avg Run Time (ms/batch): 22.284 AUC: 0.827 max AUC: 0.992

那正确吗？没有增加AUC ？？？而第一个时期之后的AUC是0.975？

Pay attention to the source code, it will be saved each time the maximum AUC is reached, so the max AUC increment is normal.

[JJY0710]

hello，because  each epoch test result will have large fluctuations, it is normal to save the largest AUC weights, which is to reflect the performance of the model. Mainly to see the training loss, the discretion loss has little impact. In fact, the best measure is to take a lot of epoch test averages.

…

------------------&nbsp;原始邮件&nbsp;------------------ 发件人:&nbsp;"davids-zhou"<[email protected]&gt;; 发送时间:&nbsp;2019年10月1日(星期二) 下午2:30 收件人:&nbsp;"samet-akcay/skip-ganomaly"<[email protected]&gt;; 抄送:&nbsp;"刮刮乐"<[email protected]&gt;;"Comment"<[email protected]&gt;; 主题:&nbsp;Re: [samet-akcay/skip-ganomaly] result confusion? (#3) the sourcecode is that weights are saved if Auc is best. so it is abnormal. train loss should be decreased. but loss of Discriminator is not changed after a while. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[DUZHUJUN]

using the code in my customize dataset，the auc can got 0.7-0.9,but the precision and recall is very low,below 10%.Do you meet the same situation?

[lzzlxxlsz]

using the code in my customize dataset，the auc can got 0.7-0.9,but the precision and recall is very low,below 10%.Do you meet the same situation?
你好，想问一下，模型训练完之后，想输入一张测试图像，加载训练好的模型，得到测试图像的重构图，怎么来进行测试呢

[Shumpei-Kikuta]

@davids-zhou and I shuffle the test data, then the result is low.

@samet-akcay @BIGBALLON I faced the same problem.
Does anyone have any idea why AUC suddenly drops when we shuffle the validation datasets?
I don't think that AUC is never affected by the order in the validation datasets.

[pankSM]

I have the same problems, do you have solutions? if so, please help me. Thanks.