[do-not-merge] Illustration how to run audio+text dataloading #9131
Conversation
Signed-off-by: Piotr Żelasko <[email protected]>
Signed-off-by: Piotr Żelasko <[email protected]>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| # elif cur_idx + tokenized_len < tgt_len: | ||
| # # Check whether the mask is applied to the correct position, the first token is turn start tokens | ||
| # if not torch.equal(target[cur_idx + 1 : cur_idx + tokenized_len], s_id[1:]): | ||
| # logging.warning("a sentence mismatches the corresponding piece " "in the conversation") |
Check notice
Code scanning / CodeQL
Commented-out code Note
| from lhotse.utils import Pathlike | ||
|
|
||
| from nemo.collections.common.data.lhotse.nemo_adapters import expand_sharded_filepaths | ||
| from nemo.collections.common.tokenizers.aggregate_tokenizer import AggregateTokenizer, TokenizerWrapper | ||
| from nemo.collections.common.tokenizers.tokenizer_spec import TokenizerSpec | ||
| from nemo.utils import logging |
Check notice
Code scanning / CodeQL
Unused import Note
| "label_start": "<extra_id_2>", | ||
| "end_of_turn": "\n", | ||
| "end_of_name": "\n", | ||
| } |
There was a problem hiding this comment.
The special tokens are tricky: perhaps we need to ensure that they are included in the tokenizer's vocab.
|
|
||
| return audio, audio_lens, tokens, token_lens, prompt_tokens, prompt_token_lens | ||
| return audio, audio_lens, tokens, token_lens, prompt_tokens, prompt_token_lens, text_minibatch |
There was a problem hiding this comment.
copy the discussion note here and some followups in the end
supports 3 different strategies of mixing speech and text data
1.current strategy (sampled together as if it's one modality by projecting duration to equivalent token counts)
2.zip sampler strategy (separate sampler for each modality with separately defined batch_duration and batch_tokens, both samplers zipped to yield a mini-batch together)
3.round-robin strategy (separate samplers as in zip but the mini-batches are alternately yielded from each)
do we expect all the three strategies will return audio and text batch separately to the model like here?
if so, what do you think about also grouping the text-audio batch data in one dict like text-only batch (and in future, add audio-audio batch)
no need to do these in this PR but can be done on top of the canary-llm branch which has a speech SFT batch compatible with mtron
There was a problem hiding this comment.
It's open for discussion. For the POC I separated text-only and audio-text portions of the mini-batch because it was easy to re-use the existing code and extend it with text-only branches. We could explore "fusing" those but we should design how the fused mini-batch would look like (e.g. do we introduce zero inputs in audio tensor for text-only examples? there will be a number of decisions to make here).
There was a problem hiding this comment.
Initially, I wanted to return sth like:
@dataclass
class MultimodalQABatch:
audio_text: dict # dict might become a nested AudioTextQABatch dataclass, similar for other fields
text_text: dict
# ... could be extend with sub fieldsBut GPT SFT model code that we inherit from expects batch to be a dict and explicitly scans it for some fields like "tokens", also discards every key that has a non-tensor value. If we want to proceed with separate dataclasses or even dicts, we would need to override some fwd/bwd functions in the parent class.
There was a problem hiding this comment.
It's open for discussion. For the POC I separated text-only and audio-text portions of the mini-batch because it was easy to re-use the existing code and extend it with text-only branches. We could explore "fusing" those but we should design how the fused mini-batch would look like (e.g. do we introduce zero inputs in audio tensor for text-only examples? there will be a number of decisions to make here).
I actually like this design. we can leave how to "fuse" to the model code to decide, e.g. model code may want to concat speech and text batch together and do one fprop instead of two
What does this PR do ?
The key config options to set are:
The values of batch_tokens, token_equivalent_duration, and quadratic_factor need tuning.
where contents of
speech_text_input.yamlare:Collection: [Note which collection this PR will affect]
Changelog
Usage
# Add a code snippet demonstrating how to use thisGitHub Actions CI
The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.
The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".
Before your PR is "Ready for review"
Pre checks:
PR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
Who can review?
Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.
Additional Information