Preventing data leakage when splitting data in setup method

[bencwallace]

The example code in What is a DataModule? randomly splits the dataset into training and validation sets as part of the setup method. When executing on multiple devices, this will generally lead to having different splits on different devices. Since (I assume) gradients are synchronized across devices, this would mean there's overlap in the data being used for training and for validation, i.e. data leakage.

Am I missing something here? What is the recommended way to avoid this issue? My guess would have been to perform the split as part of __init__ or prepare_data, but apparently that doesn't work.

[bencwallace]

Realized I posted this in the wrong place. I've moved it here but can't seem to delete it.