Self-Supervised Methods Tracker

[guarin]

The goal of this issue is to track new self-supervised methods and to which extent they are implemented in Lightly.

See also our #papers channel on discord for the newest papers!

Contributing

There are many ways on how you could contribute to either this list or the implementations :)

• Tell us about new methods in a comment
• Gather information about a new method and how we could implement it and write an issue for it
• Make a PR for a new model or parts of it
• Test already implemented methods

Methods

CrossMAE [25. Jan. 2024] Paper Code

• Status: Not Implemented

l-DAE [25. Jan. 2024] Paper

• Statue: Not Implemented

LDReg [19. Jan. 2024] Paper Code

• Status: Not implemented

AIM [16. Jan. 2024] Paper Code

• Status: 🚧 In progress
  • Missing
    •  Examples & Docs
    • Benchmark
• PR: Add AIM Model from Scalable Pre-training of Large Autoregressive Image Models #1479

SSL-EY [8. Dec. 2023] Code

• Status: 🚧 In progress
• PR: [WIP] Implement SSL-EY #1443

MMCR [3. Dec. 2023] Paper

• Status: 🚧 In progress
• Issue: Feature Request: New SSL algorithm called Maximum Manifold Capacity Representations (NeurIPS 2023) #1423

CycleCL [5. Nov. 2023] Paper

• Status: Not Implemented
• Issue: CycleCL #1447

V-JEPA [21. Sept. 2023] Paper

• Status: Not implemented

FlexPredict [31. Jul. 2023] Paper

• Status: Not implemented

EMP-SSL [08. Apr. 2023] Paper Code

• Status: 🚧 In progress
• Issue EMP-SSL #1343

MAE-CT [20. Apr. 2023] Paper Code

• Status: Not implemented

DinoV2 [14. Apr. 2023] Paper Code

• Status: Not implemented
• Issue: Add support for DinoV2 #1166

Siamese DETR [31. Mar. 2023] Paper Code

• Status: Not implemented

MixedMAE [30. Mar. 2023] Paper

• Status: Not implemented

D3SSL [29. Mar. 2023] Paper

• Status: Not implemented

Temperature Schedules for Self-Supervised Contrastive Methods on Long-Tail Data [23. Mar. 2023] Paper

• Status: 🚧 In progress
• Issue: Add SimCLR with temperature schedule #1319

All4One [16. Mar. 2023] Paper

• Status: Not implemented

LOCA [15. Mar. 2023] Paper

• Status: Not Implemented

I-JEPA [19. Jan. 2023] Paper Code

• Status: 🧪 Experimental I-JEPA #1273

SparK [9. Jan. 2023] Paper Code

• Status: Not Implemented
• Issue: Feature Request: New SSL algorithm called SparK: Sparse and Hierarchical masKed modeling #1462

CAN [20. Oct. 2022] Paper

• Status: Not implemented

PMSN [13. Oct. 2022] Paper

• Status: ✅ Done

FastSiam [22. Sept. 2022] Paper

• Status: ✅ Done

MixMAE [25. May. 2022] Paper

• Status: Not implemented

A2MIM [22. May. 2022] Paper

• Status: Not implemented
• Issue: [Suggestion] A²MIM #1315

SimMIM [18. Nov. 2021] Paper

• Status: ✅ Done
• Missing:
  • Convnet backbone SimMIM and non-ViT backbones #1135

DirectCLR [18. Oct. 2021] Paper Code

• Status: 🚧 In progress DirectCLR: Understanding Dimensional Collapse in Contrastive Self-supervised Learning #781

MoCoV3 [5. Apr. 2021] Paper Code

• Status: ✅ Done Add support for MoCo v3 #1163

SEER [2. Mar. 2021]Paper Code

• Status: Not implemented

W-MSE [13. Jul. 2020] Paper Code

• Status: ✅ Done Feature Request: W-MSE (ICML 2021) #1424
• Missing:
  • Projection head
  • Docs
  • Benchmarks

[Natyren]

Hello! If I would like to add method to lib, best way - create issue and start working on it? For example I'm ready to start implementation of MAE-CT (or MAE, looks like it isn't here).

[guarin]

Hi @Natyren! Thanks a lot for your interest. Yes please create a new issue for it. MAE is already implemented, see https://docs.lightly.ai/self-supervised-learning/examples/mae.html for examples.

[guarin]

Added FlexPredict: Predicting masked tokens in stochastic locations improves masked image modeling: https://arxiv.org/abs/2308.00566