Thoughts on OpenOOD v2

[zjysteven]

Overview

As discussed with @Jingkang50, we would like to develop an enhanced version of OpenOOD. Compared to its current version, v2 will:

• explicitly focus on OOD detection;
• improve from 1) data, 2) model architectures, 3) considered methods, and 4) impact.

(PS: Jingkang I know you said you were about to lay out a plan, but I'm free these days so I just write down something first.)

Ideas

Data

ID - OOD splits:

• MNIST: Could either keep it as is or remove it since very few papers consider MNIST nowadays.
• CIFAR-10/100: The OOD splits for CIFAR look good to me. However, we could add CIFAR-10-O and CIFAR-100-O to the near-OOD split in the same sense as ImageNet-O. CIFAR-O can be constructed from Tiny Images (the superset of CIFAR) or ImageNet32. This will require some extra effort but is very likely to bring down the performance of existing detectors.
• ImageNet-1K: I firmly think that MNIST and SVHN should be removed from the far-OOD split. Instead, Places and SUN (used in MOS) could be taken as far-OOD for ImageNet. Also the Texture one could be replaced by DTD which is more commonly used than Kylberg Texture.

Outlier data for OE-based methods:

• In general I think we need to "curate" outliers a little bit such that they don't overlap with test OOD data (in terms of semantics). A more obvious issue now is mentioned in Fixing OE training #138.

Model architectures

We definitely want to include baseline results for architectures beyond CNNs, e.g., transformers or even MLP-mixer. The caveat here is that these advanced models are often pre-trained with large-scale datasets. We may want to decouple the effect of pre-training and model architecture in our experiments.

Methods

• Include more recent methods (since mid-2022?)
• Provide better training setup (e.g., longer training, data augmentation) for stronger baselines (see "A good closed-set classifier is all you need")
• Have sanity check from the authors

Impact

• Provide an evaluation API such that given their trained models, people can easily use OpenOOD to measure OOD detection performance by simply calling a function.
• Put up a leaderboard similar to RobustBench.
• Connect OOD robustness/generalization and OOD detection community by 1) evaluating more under the full-spectrum detection setting, and 2) evaluating OOD detection performance of representative OOD generalization methods.

Discussion

There are many other things we could do, for example, measuring the inference latency of each method.

@Jingkang50 Feel free to leave comments and let others take a look and join the discussion. I don't see the due date for this year's NeurIPS dataset & benchmark track, but if it is still in June, we might be able to catch it if we can start early in February.

[malongfei01]

@zjysteven have listed several key points that I also consider to be meanningfull. I also list two points that could be discussed.

1. For training needed algorithms for ImageNet benchmark the code is run by srun,which couldn't specify gpu number and could add the logics for selecting running mode(srun or common single-process multi-GPU mode) .
2. Add hyperparameters selection.
   Next I list several points that are specifically found in ImageNet benchmark, which I think could be primarily improved for openood v2, considering its practical application value for oodd:
3. Various methods on ImageNet benchmark seem to be less efficient,especially training based methods could be carefully polished.

[Jingkang50]

Comments from @sunyiyou in our last meeting

1. Scope Extension: OOD detection to anomaly detection, uncertainty detection, OOD generalization.
2. Model Extension: small model to large model.
3. Data Extension: image to NLP, multi-modal, Graph, Time series, 3d.
4. Application Extension: spoof detection, deepfake detection.

[Jingkang50]

Hi @depshad, it is great to hear that you want to contribute! Jingyang @zjysteven is leading the API dev. Lemme ask him whether he has formed the dev team.

[zjysteven]

Hi @depshad, glad to have you. I imagine that there will be mainly two cases when using the evaluation API:

1. The user develops a new post-hoc method. In this case my current idea is to let the user provide a detection forward pass (which can output the OOD score given the input image) to our API.
2. The user develops an advanced training, and the model can be evaluated with whatever post-hoc method (e.g. MSP). In this case it would be helpful if our API can run certain post-hoc methods internally based on the model/base classifier provided by the user.

Your experience will definitely be valuable, especially in the second case. I'm currently working on setting things up for building this evaluator. I would imagine that you can start to contribute once this setup is finished. I'll keep you updated.

[depshad]

Hi @zjysteven, thank you for sharing the use cases. The plan sounds great to me. I'm looking forward to knowing more and will check here for any updates.

[depshad]

Hi @zjysteven if there's still a chance for me to contribute, please let me know :)

[zjysteven]

Hi @depshad, I sent you an invite as a collaborator earlier but you probably missed that. I've put up an evaluator class in my fork (https://github.com/zjysteven/OpenOOD) in openood/evaluation_api/evaluator.py. Below is an example.

import os, sys
sys.path.append('../')
import numpy as np
import pandas as pd
import argparse

import torch
import torch.nn as nn
import torch.nn.functional as F

from openood.evaluation_api import Evaluator

from openood.networks import ResNet18_32x32
from openood.networks.ash_net import ASHNet
from openood.networks.react_net import ReactNet
from openood.networks.conf_branch_net import ConfBranchNet
from openood.networks.godin_net import GodinNet
from openood.networks.rot_net import RotNet
from openood.networks.csi_net import CSINet
from openood.networks.udg_net import UDGNet


parser = argparse.ArgumentParser()
parser.add_argument('--root', required=True)
parser.add_argument('--postprocessor', default='msp')
parser.add_argument('--id-data', type=str, default='cifar10', choices=['cifar10', 'cifar100'])
parser.add_argument('--no-save', action='store_false', dest='save')
args = parser.parse_args()

root = args.root

# specify an implemented postprocessor
# 'msp', 'mls', 'ebo', 'odin', 'temp_scaling', 'openmax', 'vim', 'klm', 'knn', 'dice', 'gradnorm', 'mds', 'gram', 'react', 'rankfeat', 'ash'
# 'conf_branch', 'godin', 'rotpred'
postprocessor_name = args.postprocessor

if args.id_data == 'cifar10':
    num_classes = 10
elif args.id_data == 'cifar100':
    num_classes = 100
else:
    raise NotImplementedError

# load the pretrained model provided by the user
if postprocessor_name == 'conf_branch':
    net = ConfBranchNet(backbone=ResNet18_32x32(num_classes=num_classes), num_classes=num_classes)
elif postprocessor_name == 'godin':
    backbone = ResNet18_32x32(num_classes=num_classes)
    net = GodinNet(backbone=backbone,
                    feature_size=backbone.feature_size,
                    num_classes=num_classes)
elif postprocessor_name == 'rotpred':
    net = RotNet(backbone=ResNet18_32x32(num_classes=num_classes), num_classes=num_classes)
elif 'csi' in root:
    backbone = ResNet18_32x32(num_classes=num_classes)
    net = CSINet(backbone=backbone,
                    feature_size=backbone.feature_size,
                    num_classes=num_classes)
elif 'udg' in root:
    backbone = ResNet18_32x32(num_classes=num_classes)
    net = UDGNet(backbone=backbone,
                    num_classes=num_classes,
                    num_clusters=1000)
else:
    net = ResNet18_32x32(num_classes=num_classes)

net.load_state_dict(
    torch.load(os.path.join(root, 'best.ckpt'), map_location='cpu')
)
net.cuda()
net.eval()

if postprocessor_name == 'react':
    net = ReactNet(net)
elif postprocessor_name == 'ash':
    net = ASHNet(net)        

evaluator = Evaluator(
    net,
    id_name=args.id_data, # the target ID dataset
    data_root='../data',
    config_root='../configs',
    preprocessor=None, # default preprocessing
    postprocessor_name=postprocessor_name,
    postprocessor=None, # the user can pass his own postprocessor as well
    batch_size=500, # for certain methods the results can be slightly affected by batch size
    shuffle=False,
    num_workers=4
)
metrics = evaluator.eval_ood()

# metrics is a dataframe looking like this (in this case args.id_data = 'cifar10')
# just an example; the tin dataset is missing from nearood here
           FPR@95  AUROC  AUPR_IN  AUPR_OUT  CCR_4  CCR_3  CCR_2  CCR_1
cifar100    57.36  88.10    86.22     86.52   0.09   1.08   9.38  72.72
nearood     57.36  88.10    86.22     86.52   0.09   1.08   9.38  72.72
mnist       38.26  94.29    79.34     98.95   1.56   8.37  56.62  87.01
svhn        39.25  94.07    88.06     97.16   0.68   7.40  50.70  86.86
texture     48.28  91.58    93.79     85.26   0.02   1.07  17.33  81.79
places365   51.99  90.20    70.52     96.61   0.13   1.01  13.40  78.76
farood      44.45  92.53    82.93     94.50   0.60   4.46  34.51  83.60

It doesn't take much effort to build this since most of the code is already there. If you are still interested, we'd definitely appreciate it if you could take a look and give us some feedback :)