Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Scheduler should consider gradient accumulation while assigning epoch_steps? #663

Open
rohitgr7 opened this issue Apr 8, 2024 · 0 comments
Labels
type/bug Bug in code

Comments

@rohitgr7
Copy link

rohitgr7 commented Apr 8, 2024

馃悰 Bug

Here:

scheduler = get_scheduler(cfg=cfg, optimizer=optimizer, epoch_steps=epoch_steps)

Let's say total data batches are 160 and gradient accumulation is 10. Optimization step is happening only 10 times.
But here scheduler is called every time:

h2o-llmstudio/train.py

Lines 315 to 316 in a9d72ff

if scheduler is not None:
scheduler.step()

which can lead to
https://discuss.pytorch.org/t/userwarning-detected-call-of-lr-scheduler-step-before-optimizer-step-in-pytorch-1-1-0-and-later-you-should-call-them-in-the-opposite-order-optimizer-step-before-lr-scheduler-step/88295

To Reproduce

LLM Studio version

@rohitgr7 rohitgr7 added the type/bug Bug in code label Apr 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug Bug in code
Projects
None yet
Development

No branches or pull requests

1 participant