Adding spoken dialogue state tracking recipes for Spoken MultiWoz and SpokenWoz datasets

[LucasDruart]

What does this PR do?

This PR adds spoken dialogue state tracking recipes for two datasets: spoken MultiWoz and SpokenWoz. I have tried my best to conform to speechbrain's conventions. Please let me know if there are any conventions I missed. I do not know if and how recipes should be tested.

Before submitting

• Did you read the contributor guideline?
• Did you make sure your PR does only one thing, instead of bundling different changes together?
• Did you make sure to update the documentation with your changes? (if necessary)
• Did you write any new necessary tests? (not for typos and docs)
• Did you verify new and existing tests pass locally with your changes?
• Did you list all the breaking changes introduced by this pull request?
• Does your code adhere to project-specific code style and conventions?

PR review

Reviewer checklist

• Is this pull request ready for review? (if not, please submit in draft mode)
• Check that all items from Before submitting are resolved
• Make sure the title is self-explanatory and the description concisely explains the PR
• Add labels and milestones (and optionally projects) to the PR so it can be classified
• Confirm that the changes adhere to compatibility requirements (e.g., Python version, platform)
• Review the self-review checklist to ensure the code is ready for review

[asumagic]

#2441 also adds a T5 HuggingFace lobe as far as I could tell, we will see how we deal with this.

EDIT: Nevermind, it's not the same thing, thanks Adel.

[LucasDruart]

Hey @asumagic,
Thanks for taking a look into this PR, I have seen that the pre-commit pipeline fails because of the spokenwoz data preparation function which is too complicated. I believe the complexity is measured in number of lines of code, however I have trouble simplifying it since the formatting sometimes reformats into many lines. For instance line 146.

[asumagic]

Hey @asumagic, Thanks for taking a look into this PR, I have seen that the pre-commit pipeline fails because of the spokenwoz data preparation function which is too complicated. I believe the complexity is measured in number of lines of code, however I have trouble simplifying it since the formatting sometimes reformats into many lines. For instance line 146.

I think the linter measures cyclomatic complexity rather than the plain number of lines of code, so fixing formatting would probably not help. In this case, it could make sense to extract the whole for dialog_id, dialog_info in annotations.items(): loop into its own function.

As for the other pre-commit style check fails, you can fix them locally by running pre-commit run -a (pypi).

As for the other tests failing, the issue seems to be that recipe tests should be added. Recipe tests use a small subset of data and launches a dummy training for a fixed number of epochs, then asserting that some metric is respected.
(They are not run inside of CI; but rather run by SpeechBrain developers locally.)
As far as I can tell, a line should be added in tests/recipes/MultiWOZ.csv and a tests/recipes/SpokenWOZ.csv file should be created. Recipe tests are further explained in tests/recipes.

[LucasDruart]

Hi @asumagic,
I fixed the complexity and trailing space failed tests and added a test for each recipe. I had to commit without verifications the last commit since I had a failed test for large files for the sample audio (tests/samples/DST/SpokenWoz/audio_5700_train_dev/MUL1011.wav) required for the test. Let me know if I can do anything else.

[asumagic]

Now, some tests fail because:

• There are some missing docstrings.
• We have a check script that tries to ensure that training scripts use all of the YAML keys correctly. Because run.py and model.py are split, it fails to find the keys that are actually used in model.py and not run.py. Usually we just have a single, merged train.py, which should probably be done here (AFAIK Brain is not really intended for inference, only training/evaluation EDIT: to clarify, not what this PR does. what the PR does is OK IMO).

[asumagic]

BTW, I am still in the process of reviewing, but it does take some time.

[LucasDruart]

I added the missing docstrings and the missing tests.
Regarding the single train.py file, at the beginning I wanted to factorise the model between both datasets, thus separating the data pipeline from the model, but I couldn't manage to import it properly. Maybe you will have a solution for this when you get at that point in the review.
Sorry for the (many) changes as you are reviewing.

[asumagic]

Tried to review as much as I could but it remained less in-depth as I would have liked since I am not yet very familiar with this task. I have successfully run the dataset download steps and data preparation and run some steps of training so this part works OK.
I also made a number of changes you've probably noticed, let me know if they seem sane to you. The tests now pass.

There are some things I couldn't test and run (inference) as I don't think you have provided pre-trained models. If you are able to upload them, we could host it on the Dropbox and have it linked.

I would like other reviewers to also double-check the training script + metrics for correctness.

As for the duplication across training scripts (and hparams), I agree it isn't pretty, but I think this is just a flaw we have right now where very similar models just result in duplication across recipes. IMO this can be dealt with later if we figure out how to deduplicate dataset-agnostic things better.

[LucasDruart]

Sorry for the delay and thank you for your review and changes which make a lot of sense! Regarding the other points:

• I have not provided checkpoints but can provide them if needed. However it requires to re-run the whole training to reach the reported performances.
• I corrected the few naming issues and missing documentation, hope its okay now.
• I moved the Joint-Goal Accuracy tracker (used for training to keep a running JGA without storing the values) to metric_stats.py.
• I left the common part of Dialogue State Tracking evaluation in utils to be able to import it from the recipes.
• I moved the dataset specific evaluations to their recipe's meta folder.

Let me know what else I can do to go forward with this PR.