You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Technically this may not be a bug, but the current default config setting of model.ppo.num_val_samples (set to ${.num_rollout_samples}) makes it such that modifying num_rollout_samples will also change the validation set, which is bad as it may lead to deriving incorrect conclusions when checking the impact of modifying num_rollout_samples.
Expected behavior
The validation set should remain the same by default when modifying num_rollout_samples.
Additional context
My suggestion would be to deprecate num_val_samples and replace it with limit_val_batches (which is currently missing -- see #27)
The text was updated successfully, but these errors were encountered:
Describe the bug
Technically this may not be a bug, but the current default config setting of
model.ppo.num_val_samples
(set to${.num_rollout_samples}
) makes it such that modifyingnum_rollout_samples
will also change the validation set, which is bad as it may lead to deriving incorrect conclusions when checking the impact of modifyingnum_rollout_samples
.Expected behavior
The validation set should remain the same by default when modifying
num_rollout_samples
.Additional context
My suggestion would be to deprecate
num_val_samples
and replace it withlimit_val_batches
(which is currently missing -- see #27)The text was updated successfully, but these errors were encountered: