Running test code issues

[Marco-Nguyen]

Dear @yyliu01 ,

I have followed your guidelines and run your code. It can run but it has some issues:

1. When running the test for Fishyscapes_ls dataset, it finished the iteration but it did not give any results.
   

2. When running final_test_inlier, I had the following error: AttributeError: module 'collections' has no attribute 'Iterable'. I have installed packages according to requirements.txt

Can you give me some guidance on how to solve those issues?

Thank you in advance.

[yyliu01]

Hi @Marco-Nguyen ,

1). In fishyscapes l&f, It would take long time to calculate the final measurements (e.g., FPR, AuPRC) after the inference. Please give it a few minutes and don't interrupt it.

2). I think it should refer to the incompatibilities of different python versions. Could you please follow the guide in this page? Alternatively, this link might be what you are looking for.

Cheers,
Yuyuan

[Marco-Nguyen]

@yyliu01 for the 1st question, I did not interrupt, the code stops itself immediately after the progress bar reaches the end. I wonder why it happens. I am running my code on Google Colab. I tried to run the test for Fishyscapes_ls separately and it still stops itself

[yyliu01]

Hi @Marco-Nguyen ,

Sorry for the misunderstanding, because I saw a "^C" in the screenshot you uploaded before.

The potential reason is the OOM, as the program saves all the pixel-wise prediction results and calculates the final metrics with one time. Could you monitor the memory use based on the shell command "htop"?

If that is the case, the potential solutions are:
1). utilising sample-wise evaluation via this function, or

2). utilising the evaluation from Meta-OoD in here.

Cheers,
Yuyuan

[Marco-Nguyen]

Hi @yyliu01 , I have successfully executed the code without any error. I have some questions.

1. Does the code support inferencing on a new image (not in the validation datasets) produce the result and then visualize it?
2. Is there an option to visualize the results of validation sets beside wandb?

[Marco-Nguyen]

And another question @yyliu01 , I don't know whether should I raise another issue. But when running the code, it does not reproduce the reported results in the paper for all validation datasets, except for Fishyscapes static. Here is my reported result after running RPL + CoroCL test:

I ran the test on Google Colab with Tesla T4 GPU. Could you point out what might go wrong?

[yyliu01]

Hi @Marco-Nguyen,

I apologize for the late reply. Please open a new issue or reopen the current issue so that I can notice your question earlier next time.

1). The reported results in SMIYC benchmark are based on the official code. Comparing with our evaluation function (which is only for validation purpose), it contains an additional post-process to fine-tuning the final results.

2). The potential reason of the different results from other benchmark is the different validation between the image-wise function and the statistic function, while all the results from the paper are based on the statistic manner.

I have retested the provided rpl_corocl ckpt "rev3," and the results are the same as we reported in the paper, except for SMIYC, which is based on the official evaluation. Please feel free to evaluate the provided ckpts based on their code, and I attached the report based on our valid function below:

[iccv23][INFO] validating segment_me_anomaly dataset with energy ...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:04<00:00, 2.13it/s]
[iccv23][CRITICAL] AUROC score for segment_me_anomaly: 0.9806
[iccv23][CRITICAL] AUPRC score for segment_me_anomaly: 0.8854
[iccv23][CRITICAL] FPR@TPR95 for segment_me_anomaly: 0.0722
[iccv23][INFO] validating segment_me_obstacle dataset with energy ...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:17<00:00, 1.71it/s]
[iccv23][CRITICAL] AUROC score for segment_me_obstacle: 0.9996
[iccv23][CRITICAL] AUPRC score for segment_me_obstacle: 0.9672
[iccv23][CRITICAL] FPR@TPR95 for segment_me_obstacle: 0.0010
[iccv23][INFO] validating Fishyscapes_static dataset with energy ...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:17<00:00, 1.69it/s]
[iccv23][CRITICAL] AUROC score for Fishyscapes_static: 0.9973
[iccv23][CRITICAL] AUPRC score for Fishyscapes_static: 0.9247
[iccv23][CRITICAL] FPR@TPR95 for Fishyscapes_static: 0.0085
[iccv23][INFO] validating Fishyscapes_ls dataset with energy ...
100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:53<00:00, 1.88it/s]
[iccv23][CRITICAL] AUROC score for Fishyscapes_ls: 0.9939
[iccv23][CRITICAL] AUPRC score for Fishyscapes_ls: 0.7061
[iccv23][CRITICAL] FPR@TPR95 for Fishyscapes_ls: 0.0252
[iccv23][INFO] validating road_anomaly dataset with energy ...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 60/60 [00:13<00:00, 4.43it/s]
[iccv23][CRITICAL] AUROC score for road_anomaly: 0.9572
[iccv23][CRITICAL] AUPRC score for road_anomaly: 0.7161
[iccv23][CRITICAL] FPR@TPR95 for road_anomaly: 0.1774

PLEASE NOTE: You can also observe the same record in 9768 iteration of our provided training log.

---

Update

I've just remembered that I've uploaded the SMIYC validation results based on the official code in this link before.

[Marco-Nguyen]

Thanks for the response. I will have a look and get back to you

[yyliu01]

Very welcome, @Marco-Nguyen.
Just open a new issue if you have more questions and I will see it in first time.

[skming666]

文章中的anomaly map 是怎么生成的，我在代码中没看到可视化内容

[yyliu01]

@skming666 是用energy的结果归一化之后生成score, 然后喂给seaborn.heatmap进行的可视化。如果还有其他问题, 麻烦请单独再开issue, 谢谢。