[App]: Seeding a new sweep with previous runs has an implicit limit of 1000 previous runs

[RylanSchaeffer]

Current Behavior

I will select a number of runs (e.g., 3420 runs from two previous sweeps) to seed a new sweep, but W&B will select a subset (specifically, 1000 runs) for the new sweep. This is new behavior; W&B didn't use to behave as such.

Expected Behavior

I expect that when I seed a new sweep with > 1000 runs, the sweep will be seeded with the correct number of runs and will not truncate to 1000.

Steps To Reproduce

No response

Screenshots

Environment

OS: Ubuntu 20.04.4 LTS

Browsers:

Version: wandb, version 0.16.5

Additional Context

No response

[ArtsiomWB]

Hi Rylan, apologies you are seeing this behavior on your side! Could you please point me to a workspace where you are experiencing this behavior?

[RylanSchaeffer]

I would prefer to not publicly post links 😬

Is there some way I can privately communicate such information to you?

[RylanSchaeffer]

Ehh I care more about having a quick solution.

Here is a previous sweep with 1140 successful runs: https://wandb.ai/rylan/signal_prop/sweeps/ejkla2x2

I want to use all 1140 finished runs to seed a new sweep that will also include a second value of one hyperparameter (the previous sweep has only mu=1.0, now I want to to sweep mu=1.0 and mu=0.0):

When I initialize a new sweep, it has only 1000 previous runs: https://wandb.ai/rylan/signal_prop/sweeps/n5nhqi8e/table?nw=nwuserrylan

[ArtsiomWB]

Hi @RylanSchaeffer, when you start a new sweep up from a previous sweep, do you do it using the create a sweep button from the runs table tab? I will try to repro this behavior on my side.

[RylanSchaeffer]

That's correct. I filter for runs from the previous sweep(s) and then use the Create a Sweep button. I've done this for over a year now and it used to work as intended. This 1000 previous runs cap is new, as best as I can tell.

[ArtsiomWB]

When you initialize the sweep, how many runs do you see next to the configure prior runs button?

[ArtsiomWB]

Hi there, I wanted to follow up on this request. Please let us know if we can be of further assistance or if your issue has been resolved.

[RylanSchaeffer]

Here I am selecting 1140 runs:

I then press "Create Sweep". I specify additional values for a hyperparameter mu such that the finished sweep will be a strict subset of the new sweep:

I then press "Initialize Sweep". If I then navigate to the new sweep page, there are 1000 prior runs, even though I selected 1140 runs.

I hope this helps!

[ArtsiomWB]

Thank you so much for sending over all of the screenshots, this helps a ton. I checked internally and this is a known bug. I have went ahead and re-escalated the issue and our engineering and product teams will look into it.

Apologies for the inconvenience. If we have any follow up questions regarding this behavior, we will put them in here.

[RylanSchaeffer]

Awesome thank you! If you could pass on a message, this has really hampered the utility I get from W&B sweeps. I oftentimes want to sweep a subset of hyperparameter space to get specific results quickly, then expand that subset to get additional results more slowly. I can now no longer do this. This was one of the top utilities of W&B sweeps for me personally

[ArtsiomWB]

Totally will! By the way, How long ago were you able to create sweeps from over 1k runs? I do see that the behavior you are currently experiencing is a long-standing bug that was marked a back logged because our engineering team could not reproduce it before.