-
Notifications
You must be signed in to change notification settings - Fork 607
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can DALI be integrated into HuggingFace Trainer? #5421
Comments
I just checked and unfortunately, HuggingFace trainer (I checked only one example so far) expects PyTorch DataIterator (checks for the type of passes object). I still believe it is possible to make DALI work there but I may need a bit of adjustments to the internal methods. If you have some spare time we would be more than happy to review a PR with such an example. |
Thanks for your reply. Yeah, I have tried integrating DALI into the HuggingFace trainer. As you mentioned, it seems to expect either torch.utils.data.Dataset or torch.utils.data.IterableDataset as input. It might be possible to customize a data collator, which can preprocess data in a DALI pipeline, for the HuggingFace trainer, but I'm not sure if it will work. If you have any idea for this, please let me know. I'm sure it would be a huge step for the whole community! |
I did some basic investigation:
|
For the third point, get_train_dataloader returns self.accelerator.prepare(DataLoader(train_dataset, **dataloader_params)) for a distributed training or evaluation. So what should I do in this setting? Hope for your help. |
@ShyFoo - I'm afraid I don't know the answer. My idea was to avoid wrapping DALI into DataLoader as it serves as one already. If you have time I would be more than happy to learn your findings. |
Got it. I have tried using DALI + DeepSpeed, where I have to define a triaing loop and other practical functions myself, instead of using the Huggingface Trainer for convenience, so I prefer to use DALI, DeepSpeed, and Huggingface Trainer simultaneously if possible. |
If you have any examples of DALI + DeepSpeed feel free to post them as PR to DALI - we would be more than happy to enrich our documentation and the base of samples. |
Well, I'd be happy to. |
Describe the question.
Is it possible to use DALI as the dataloader for HuggingFace Trainer?
Check for duplicates
The text was updated successfully, but these errors were encountered: