[Q about Ludwig + DeepLearning.ai: Efficient Fine-Tuning for Llama2-7b on a Single GPU] Finetune seems not working

[rzsgrt]

Hi, I come here because event you create with deeplearning.ai. Thank you for sharing with us about ludwig and predibase.
But I have question about the result. So I just run it without any changes on param.
Using test examples, 4/5 output just repeating instruction and 1 repeat the answer. Is it normal?
My assumption is I need to increase epochs. Does anyone try to train with higher epoch? How many epoch you guys get reasonable answer?
Thank you!

[tgaddair]

Hey @rzsgrt, the first change that would be needed to improve on the example notebook results is to increase the number of examples in the dataset. In the notebook, we only selected 100 examples to make the demo run quickly for first-time users. But if you want to improve performance, I would start by using the full dataset.

[tgaddair]

Hey @rzsgrt, we discovered an issue with the learning rate after changing some of the default params in Ludwig v0.8.2. Can you take another look at the now updated notebook and try again?

[rzsgrt]

Hi @tgaddair, thanks for your answer. I notice learning rate param you change. Also, I try to increase train data up to 500. After that result quite good. But when I try train 1000 example, it start oom. Is it because we put all data at gpu?

[tgaddair]

Hi @rzsgrt, the issue there appears to be related to the length of one of the sequences in the dataset, which pushes the memory above the limit. One quick fix would be to set max_sequence_length: 256 or a similar value (to truncate the inputs). We're also working on something on the optimizer side we hope to land next week that will reduce memory pressure during training.

[rzsgrt]

I see, so we reducing matrix size total data train x sequence_length. I don't know the detail, do we push all train data on gpu at first?

[tgaddair]

We push all the training data in a single batch (determined by the param batch_size) to the GPU at once. But the rest of the data resides in CPU memory until it's needed to avoid out-of-memory errors.

[rzsgrt]

I see. thank you so much for your answer. I'll close this discussion