Renting CPUs -

[georgedei]

When renting GPUs we have a choice of multiple GPUs. Does the VRAM combine to make training with max_length 800 possible? Or is the VRAM limit per each GPU?

• sorry I meant GPU, not CPU

[martinambrus]

The VRAM doesn't combine to the best of my knowledge, since a GPU needs to have all data for a single WAV file for it to procees it as a whole (if I'm wrong, please someone correct me). That'd be why we need to play with number of batches and max_len settings to fit the training into our GPUs.

That said, I successfully used max_len set to 1760 on a A100 with 84GB VRAM, so 800 max_len should not be a problem to train, if you have the VRAM.

[georgedei]

Thanks Martin, what has been your experience with using max_length 1760, how does it affect the model? Is the model more capable of capturing nuances and emotional content?
Thanks for you detailed descriptions of your training process, this is very helpful.
Specifically, I'd like to understand the effect of max_length on model quality. It has become pretty difficult to find A100 machines to rent. The A6000 is more available.

[martinambrus]

I only trained with a short max_len of 200 before due to the constrains of Google Colab, so 1760 definitelly improves on that :))) But I haven't tried any other max_len setting really (like 400 or 800), since I wanted the model to be trained on all the material I have. So this is all I have without anything production-ready to compare it to.

That said, the model quality is definitelly better with larger max_len (at least from my understanding) because then even more complex sentences can be process and learned with more complex emotional background.

And yes, I can see that there is a shortage of A100 machines on vast.ai - which wasn't the case at all when I tried. But those already expired, so I guess nobody re-added them (which is a shame). Maybe other GPU providers will have them available.

[lpscr]

hi @martinambrus How can I train the first stage with multiple GPUs? I tried renting 2 A6000s for testing, but it didn't start; it got stuck. When I use multiple GPUs, it freezes and doesn't continue However, when I use just one GPU, it works fine. What am I doing wrong?
i use vast.ai

i use command

accelerate config

In which compute environment are you running? ([0] This machine, [1] AWS (Amazon SageMaker)): 0
Which type of machine are you using? ([0] No distributed training, [1] multi-CPU, [2] multi-GPU, [3] TPU): 2
How many different machines will you use (use more than 1 for multi-node training)? [1]: 1
Do you want to use DeepSpeed? [yes/NO]: no
Do you want to use FullyShardedDataParallel? [yes/NO]: no
How many GPU(s) should be used for distributed training? [1]: 2
Do you wish to use FP16 or BF16 (mixed precision)? [NO/fp16/bf16]: no

accelerate launch train_first.py --config_path ./Configs/config.yml

and freeze say something about the kernel only working when i run like this but i have 2 A6000
Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.

accelerate launch --num_processes 1 train_first.py --config_path ./Configs/config.yml
say again about the kernel and continue and train working fine but only one gpu :(

can you please tell me how you make work ?

[martinambrus]

This never happened to me, neither I ever ran accelerate config while training. This is the command I used for training: accelerate launch --mixed_precision=fp16 train_first.py --config_path ./Configs/config.yml.

And yes, I did do this on multiple A6000 instances, so it's bound to work. Not sure why it wouldn't in your case but I don't have your data files, so if this still hangs even after trying the command I used, perhaps it's the data fault.

How many WAV files do you have? And did you filter out all files shorter than 1s? That could also bring some trouble if you have some of those.

Also - I assume you're training single-language model? I only did that, never tried multi-lang models myself.

[lpscr]

@martinambrus Thank you very much for your reply! So, I only need to run this command

accelerate launch --mixed_precision=fp16 train_first.py --config_path ./Configs/config.yml
When you run this command, does it start right away or do you need to wait for a few minutes?

Good to know. I'll try it again then maybe something wrong ;)

[martinambrus]

For me, that command basically starts right away. No waiting a few minutes.