Loss calculation always 0

[sanipanwala]

Hello,

I'm trying to fine-tune the 34B model but during fine-tuning, I always get a loss 0. While I was able to fine-tune 7B and 13B models but not 34B.

Let me know if I'm overlooking this or please give me suggestions.

Thanks.

[jgehring]

Hi @sanipanwala, we don't provide support for fine-tuning in this repository. Which tools are you using for this? Are you sure they support the 34B model well? The exact same setting works for 7B and 13B? In any case, a loss of 0 at the start of training is a good indication that something's going wrong.

[sanipanwala]

@jgehring I mean I'm using "codellama/CodeLlama-34b-hf" model and running a normal Python script and yes same configuration works with 7B and 13B.

Thanks.

[sssszh]

@sanipanwala
Hi, have you solved this problem yet?

I found the same problem when trying to peft fine-tune CodeLLama-7B (using LlamaForSequenceClassification), the Loss is always 0 during the fine-tuning.

Thanks！

[sanipanwala]

Hi @sssszh ,

No, I haven't found any solution yet.

Thanks,
Sani