Support multiple validation datasets when dataloader_persistent_workers=True

[bastienlc]

What does this PR do?

Fixes #30527

This PR adds support for multiple validation datasets in transformers.Trainer when dataloader_persistent_workers=True. The current behavior is that all validations will be done on the first validation dataset.

I had to pass the validation_dataset_name to the evaluate and get_eval_dataloader methods so that the right validation dataloader could be retrieved.

Who can review?

@muellerzr

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[muellerzr]

Hi! Terribly sorry for missing this.

From an API perspective I think we can be even simpler than this however.

What if we set eval_dataset to be an Optional[Dataset, str] and specify how a str dataset key is used/passed. I don't think we need to go as far as a new argument. WDYT?

cc @amyeroberts also for thoughts.

And terribly sorry this went under the radar for so long, I appreciate the ping in the issue

[amyeroberts]

@muellerzr I like the sound of this solution :)

[bastienlc]

Hi, sure that would be better, something like what I just pushed ?

Or maybe we'd be better off hashing the evaluation dataset like datasets does for caching ? That would avoid having to check multiple cases.

[amyeroberts]

@bastienlc The current state LGTM. Let's get @muellerzr's opinion though - as he know's the trainer better than me :)

[muellerzr]

Thanks for the refactor! Change indeed looks good to me here, can you rebase from main so we can check that the CI is ✅ ?

[muellerzr]

Great! cc @amyeroberts for final review :)

[amyeroberts]

Thanks for adding!

Only thing to add is a test to make sure evaluate behaves as expected for the different types of eval_dataset