Why changing logger arguments affects checkpoint directory?

[Yevgnen]

I noticed when I change the project line in logger/wandb.yaml to project: "xxx" and comment the dirpath: "checkpoints/" line out in callbacks/default.yaml, the checkpoint directory changes to xxx/2p9w1mmv/checkpoints. Although keeping dirpath: "checkpoints/" seems OK, but it currently does not work with DDP.

├── config_tree.log
├── train.log
├── wandb
│   ├── debug-internal.log -> run-20220222_112915-2p9w1mmv/logs/debug-internal.log
│   ├── debug.log -> run-20220222_112915-2p9w1mmv/logs/debug.log
│   ├── latest-run -> run-20220222_112915-2p9w1mmv
│   └── run-20220222_112915-2p9w1mmv
│       ├── files
│       │   ├── config.yaml
│       │   ├── output.log
│       │   ├── requirements.txt
│       │   ├── wandb-metadata.json
│       │   └── wandb-summary.json
│       ├── logs
│       │   ├── debug-internal.log
│       │   └── debug.log
│       ├── run-2p9w1mmv.wandb
│       └── tmp
│           └── code
└── xxx
    └── 2p9w1mmv
        └── checkpoints
            ├── epoch_000.ckpt
            └── last.ckpt

10 directories, 14 files

[ashleve]

As you can see from many issues, multirun + DDP currently doesn't work correctly. This is due to the way hydra creates a new work dir for every run, which is not compatible with DDP which runs new process for each GPU.

A while ago I made an issue about in lightning repo, read it if you want to learn more: Lightning-AI/pytorch-lightning#11300

I don't see anything we can do about it for now. It's being worked on, so the future release might fix it.