Skip to content

chingyaoc/DCL

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

14 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Debiased Contrastive Learning

A prominent technique for self-supervised representation learning has been to contrast semantically similar and dissimilar pairs of samples. Without access to labels, dissimilar (negative) points are typically taken to be randomly sampled datapoints, implicitly accepting that these points may, in reality, actually have the same label. Perhaps unsurprisingly, we observe that sampling negative examples from truly different labels improves performance, in a synthetic setting where labels are available. Motivated by this observation, we develop a debiased contrastive objective that corrects for the sampling of same-label datapoints, even without knowledge of the true labels.

Debiased Contrastive Learning NeurIPS 2020 [paper]
Ching-Yao Chuang, Joshua Robinson, Lin Yen-Chen, Antonio Torralba, and Stefanie Jegelka

Prerequisites

  • Python 3.7
  • PyTorch 1.3.1
  • PIL
  • OpenCV

Contrastive Representation Learning

We can train standard (biased) or debiased version (M=1) of SimCLR with main.py on STL10 dataset.

flags:

  • --debiased: use debiased objective (True) or standard objective (False)
  • --tau_plus: specify class probability
  • --batch_size: batch size for SimCLR

For instance, run the following command to train a debiased encoder.

python main.py --tau_plus = 0.1

*Due to the implementation of nn.DataParallel(), training with at most 2 GPUs gives the best result.

Linear evaluation

The model is evaluated by training a linear classifier after fixing the learned embedding.

path flags:

  • --model_path: specify the path to saved model
python linear.py --model_path results/model_400.pth

Pretrained Models

tau_plus Arch Latent Dim Batch Size Accuracy(%) Download
Biased tau_plus = 0.0 ResNet50 128 256 80.15 model
Debiased tau_plus = 0.05 ResNet50 128 256 81.85 model
Debiased tau_plus = 0.1 ResNet50 128 256 84.26 model

Citation

If you find this repo useful for your research, please consider citing the paper

@article{chuang2020debiased,
  title={Debiased contrastive learning},
  author={Chuang, Ching-Yao and Robinson, Joshua and Lin, Yen-Chen and Torralba, Antonio and Jegelka, Stefanie},
  journal={Advances in Neural Information Processing Systems},
  volume={33},
  year={2020}
}

For any questions, please contact Ching-Yao Chuang ([email protected]).

Acknowledgements

Part of this code is inspired by leftthomas/SimCLR.

About

NeurIPS 2020, Debiased Contrastive Learning

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages