Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bioconda-utils autobump --create-pr crashes, but adding --no-shuffle fixes it #961

Open
niemasd opened this issue Mar 7, 2024 · 0 comments
Open

bioconda-utils autobump --create-pr crashes, but adding --no-shuffle fixes it #961

niemasd opened this issue Mar 7, 2024 · 0 comments

Comments

@niemasd
Copy link

niemasd commented Mar 7, 2024

I'm trying to autobump my viral_consensus recipe using the "Updating recipes" guide, and when I ran the following:

bioconda-utils autobump recipes/ config.yml --packages viral_consensus --create-pr

it crashed as follows:

08:14:08 BIOCONDA INFO Hosters loaded: ['FTPHoster', 'GithubRelease', 'GithubTag', 'GithubReleaseAttachment', 'GithubRepoStore', 'Bioconductor', 'CargoPort', 'SourceForge', 'PyPi', 'Bioarchive', 'CPAN', 'CRAN', 'BitBucketTag', 'BitBucketDownload', 'GitlabTag']
08:14:08 BIOCONDA WARNING Selected 1 packages
08:14:08 BIOCONDA INFO Building Recipe DAG
Loading Recipes...: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 9289/9289 [00:31<00:00, 298.97it/s]
08:14:39 BIOCONDA INFO Building Recipe DAG: done (8565 nodes, 28466 edges)
08:14:39 BIOCONDA WARNING Graph contains 4 packages (blacklist excluded)
08:14:40 BIOCONDA WARNING Excluding blacklisted recipes
08:14:40 BIOCONDA WARNING Checking out master
08:14:40 BIOCONDA INFO Updating master to latest project master
08:14:40 BIOCONDA INFO Updating and pruning remotes
08:15:19 BIOCONDA INFO Loading package lists for ['conda-forge']
defaults/noarch: 5.03MB [00:00, 31.2MB/s]                                                                                          | 0/9 [00:00<?, ?files/s]
defaults/osx: 44.4MB [00:01, 29.2MB/s]                                                                                     | 1/9 [00:00<00:03,  2.47files/s]
defaults/linux: 48.4MB [00:03, 15.2MB/s]█▎                                                                                 | 2/9 [00:03<00:13,  1.97s/files]
bioconda/linux: 35.5MB [00:03, 10.7MB/s]█████████████                                                                      | 3/9 [00:03<00:07,  1.29s/files]
conda-forge/noarch: 82.8MB [00:04, 17.9MB/s]████████████████████▋                                                          | 4/9 [00:04<00:04,  1.01files/s]
bioconda/osx: 28.4MB [00:02, 12.8MB/s]██████████████████████████████████████▎                                              | 5/9 [00:06<00:05,  1.48s/files]
bioconda/noarch: 36.4MB [00:00, 64.0MB/s]███████████████████████████████████████████████                                   | 6/9 [00:07<00:03,  1.16s/files]
conda-forge/osx: 198MB [00:10, 19.8MB/s]███████████████████████████████████████████████████████████▋                       | 7/9 [00:09<00:03,  1.62s/files]
conda-forge/linux: 234MB [00:15, 15.5MB/s]█████████████████████████████████████████████████████████████████████▎           | 8/9 [00:16<00:03,  3.14s/files]Downloading: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:22<00:00,  2.49s/files]
08:15:43 BIOCONDA INFO Running pipeline with these steps:
08:15:43 BIOCONDA INFO  1. Exclude recipes disabled via config
08:15:43 BIOCONDA INFO  2. Exclude blacklisted recipes
08:15:43 BIOCONDA INFO  3. Exclude sub-recipes
08:15:43 BIOCONDA INFO  4. Exclude recipes depending on packages in need of update
08:15:43 BIOCONDA INFO  5. Load recipe from git (honoring active branches)
08:15:43 BIOCONDA INFO  6. Exclude recipes in conda-forge
08:15:43 BIOCONDA INFO  7. Bump recipes in need of rebuild after pinning changes
08:15:43 BIOCONDA INFO  8. Scan upstream for new releases and update recipe
08:15:43 BIOCONDA INFO  9. Update source checksums
08:15:43 BIOCONDA INFO  10. Write recipe to per-recipe git branch
08:15:43 BIOCONDA INFO  11. Create or Update PR on GitHub
08:15:45 BIOCONDA ERROR Dropping into debugger
Traceback (most recent call last):
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/cli.py", line 78, in wrapper
    func(*args, **kwargs)
  File "<boltons.funcutils.FunctionBuilder-19>", line 2, in autobump
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/cli.py", line 97, in wrapper
    func(*args, **kwargs)
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/cli.py", line 1021, in autobump
    scanner.run()
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/autobump.py", line 204, in run
    res = super().run()
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/aiopipe.py", line 126, in run
    self.loop.run_until_complete(task)
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/autobump.py", line 225, in _async_run
    return await super()._async_run()
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/aiopipe.py", line 163, in _async_run
    await self.queue_items(source_q, return_q)
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/autobump.py", line 217, in queue_items
    await self.recipe_source.queue_items(send_q, return_q)
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/autobump.py", line 147, in queue_items
    random.shuffle(remaining_recipes)
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/random.py", line 307, in shuffle
    x[i], x[j] = x[j], x[i]
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/networkx/classes/reportviews.py", line 194, in __getitem__
    return self._nodes[n]
KeyError: 0
  0%|                                                                                                                                 | 0/4 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/home/niema/miniconda3/envs/bioconda/bin/bioconda-utils", line 10, in <module>
    sys.exit(main())
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/cli.py", line 1114, in main
    argh.dispatch_commands([
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/argh/dispatching.py", line 328, in dispatch_commands
    dispatch(parser, *args, **kwargs)
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/argh/dispatching.py", line 174, in dispatch
    for line in lines:
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/argh/dispatching.py", line 277, in _execute_command
    for line in result:
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/argh/dispatching.py", line 260, in _call
    result = function(*positional, **keywords)
  File "<boltons.funcutils.FunctionBuilder-22>", line 2, in autobump
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/cli.py", line 137, in wrapper
    func(*args, **kwargs)
  File "<boltons.funcutils.FunctionBuilder-21>", line 2, in autobump
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/cli.py", line 66, in wrapper
    func(*args, **kwargs)
  File "<boltons.funcutils.FunctionBuilder-20>", line 2, in autobump
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/cli.py", line 78, in wrapper
    func(*args, **kwargs)
  File "<boltons.funcutils.FunctionBuilder-19>", line 2, in autobump
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/cli.py", line 97, in wrapper
    func(*args, **kwargs)
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/cli.py", line 1021, in autobump
    scanner.run()
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/autobump.py", line 204, in run
    res = super().run()
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/aiopipe.py", line 126, in run
    self.loop.run_until_complete(task)
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/autobump.py", line 225, in _async_run
    return await super()._async_run()
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/aiopipe.py", line 163, in _async_run
    await self.queue_items(source_q, return_q)
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/autobump.py", line 217, in queue_items
    await self.recipe_source.queue_items(send_q, return_q)
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/autobump.py", line 147, in queue_items
    random.shuffle(remaining_recipes)
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/random.py", line 307, in shuffle
    x[i], x[j] = x[j], x[i]
  File "/home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/networkx/classes/reportviews.py", line 194, in __getitem__
    return self._nodes[n]
KeyError: 0
08:15:49 ASYNCIO ERROR Task was destroyed but it is pending!
task: <Task pending name='Task-54' coro=<AsyncPipeline.worker() running at /home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/aiopipe.py:188> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f323b95eee0>()]>>
08:15:49 ASYNCIO ERROR Task was destroyed but it is pending!
task: <Task pending name='Task-53' coro=<AsyncPipeline.worker() running at /home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/aiopipe.py:188> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f323b95e7f0>()]>>
08:15:49 ASYNCIO ERROR Task was destroyed but it is pending!
task: <Task pending name='Task-52' coro=<AsyncPipeline.worker() running at /home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/aiopipe.py:188> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f323b95e0a0>()]>>
08:15:49 ASYNCIO ERROR Task was destroyed but it is pending!
task: <Task pending name='Task-51' coro=<AsyncPipeline.worker() running at /home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/aiopipe.py:188> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f323b95e9a0>()]>>
08:15:49 ASYNCIO ERROR Task was destroyed but it is pending!
task: <Task pending name='Task-50' coro=<AsyncPipeline.worker() running at /home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/aiopipe.py:188> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f323b95e970>()]>>
08:15:49 ASYNCIO ERROR Task was destroyed but it is pending!
task: <Task pending name='Task-49' coro=<AsyncPipeline.worker() running at /home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/aiopipe.py:188> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f323b95e7c0>()]>>
08:15:49 ASYNCIO ERROR Task was destroyed but it is pending!
task: <Task pending name='Task-48' coro=<AsyncPipeline.worker() running at /home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/aiopipe.py:188> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f323b95eb80>()]>>
08:15:49 ASYNCIO ERROR Task was destroyed but it is pending!
task: <Task pending name='Task-47' coro=<AsyncPipeline.worker() running at /home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/aiopipe.py:188> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f323b95e820>()]>>
08:15:49 ASYNCIO ERROR Task was destroyed but it is pending!
task: <Task pending name='Task-46' coro=<AsyncPipeline.show_progress() running at /home/niema/miniconda3/envs/bioconda/lib/python3.8/site-packages/bioconda_utils/aiopipe.py:180> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f323b95e1f0>()]>>
  0%|

Surprisingly, the crash was occurring during random.shuffle (not sure why). I then added the --no-shuffle argument:

bioconda-utils autobump recipes/ config.yml --packages viral_consensus --create-pr --no-shuffle

and it didn't crash:

08:23:20 BIOCONDA INFO Hosters loaded: ['FTPHoster', 'GithubRelease', 'GithubTag', 'GithubReleaseAttachment', 'GithubRepoStore', 'Bioconductor', 'CargoPort', 'SourceForge', 'PyPi', 'Bioarchive', 'CPAN', 'CRAN', 'BitBucketTag', 'BitBucketDownload', 'GitlabTag']
08:23:20 BIOCONDA WARNING Selected 1 packages
08:23:20 BIOCONDA INFO Building Recipe DAG
Loading Recipes...: 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 9289/9289 [00:31<00:00, 290.84it/s]
08:23:52 BIOCONDA INFO Building Recipe DAG: done (8565 nodes, 28466 edges)
08:23:52 BIOCONDA WARNING Graph contains 4 packages (blacklist excluded)
08:23:52 BIOCONDA WARNING Excluding blacklisted recipes
08:23:52 BIOCONDA WARNING Checking out master
08:23:53 BIOCONDA INFO Updating master to latest project master
08:23:53 BIOCONDA INFO Updating and pruning remotes
08:23:55 BIOCONDA INFO Loading package lists for ['conda-forge']
defaults/noarch: 5.03MB [00:00, 17.9MB/s]                                                                                          | 0/9 [00:00<?, ?files/s]
bioconda/linux: 35.5MB [00:01, 28.3MB/s]                                                                                   | 1/9 [00:00<00:06,  1.26files/s]
defaults/linux: 48.4MB [00:02, 25.4MB/s]█▎                                                                                 | 2/9 [00:02<00:08,  1.27s/files]
defaults/osx: 44.4MB [00:04, 11.5MB/s]███████████████                                                                      | 3/9 [00:04<00:09,  1.60s/files]
bioconda/osx: 28.4MB [00:00, 35.5MB/s]██████████████████████████▋                                                          | 4/9 [00:04<00:05,  1.19s/files]
conda-forge/noarch: 82.8MB [00:06, 14.1MB/s]████████████████████████████████▎                                              | 5/9 [00:05<00:04,  1.02s/files]
bioconda/noarch: 36.4MB [00:04, 9.43MB/s]███████████████████████████████████████████████                                   | 6/9 [00:09<00:06,  2.06s/files]conda-forge/osx: 198MB [00:10, 19.0MB/s]███████████████████████████████████████████████████████████▋                       | 7/9 [00:10<00:03,  1.61s/files]
conda-forge/linux: 234MB [00:16, 14.5MB/s]█████████████████████████████████████████████████████████████████████▎           | 8/9 [00:16<00:03,  3.10s/files]
Downloading: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:23<00:00,  2.64s/files]
08:24:20 BIOCONDA INFO Running pipeline with these steps:
08:24:20 BIOCONDA INFO  1. Exclude recipes disabled via config
08:24:20 BIOCONDA INFO  2. Exclude blacklisted recipes
08:24:20 BIOCONDA INFO  3. Exclude sub-recipes
08:24:20 BIOCONDA INFO  4. Exclude recipes depending on packages in need of update
08:24:20 BIOCONDA INFO  5. Load recipe from git (honoring active branches)
08:24:20 BIOCONDA INFO  6. Exclude recipes in conda-forge
08:24:20 BIOCONDA INFO  7. Bump recipes in need of rebuild after pinning changes
08:24:20 BIOCONDA INFO  8. Scan upstream for new releases and update recipe
08:24:20 BIOCONDA INFO  9. Update source checksums
08:24:20 BIOCONDA INFO  10. Write recipe to per-recipe git branch
08:24:20 BIOCONDA INFO  11. Create or Update PR on GitHub
  0%|                                                                                                                                 | 0/4 [00:00<?, ?it/s]Updating build index: /home/niema/miniconda3/envs/bioconda/conda-bld

08:24:26 BIOCONDA INFO htslib had no changes
08:24:26 BIOCONDA INFO Recipe viral_consensus: updating from remote bump/viral_consensus
 75%|██████████████████████████████████████████████████████████████████████████████████████████▊                              | 3/4 [00:04<00:01,  1.32s/it]Updating build index: /home/niema/miniconda3/envs/bioconda/conda-bld

08:24:29 BIOCONDA INFO Updating checksum for viral_consensus 0.0.5
08:24:35 BIOCONDA INFO Checking out branch bump/viral_consensus
08:24:36 BIOCONDA INFO viral_consensus has update in progress
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:14<00:00,  3.51s/it]
08:24:36 BIOCONDA WARNING Finished update
08:24:36 BIOCONDA INFO Unrecognized URL stats:
08:24:36 BIOCONDA INFO
08:24:36 BIOCONDA INFO Recipe status statistics:
08:24:36 BIOCONDA INFO IsSubRecipe: 2
08:24:36 BIOCONDA INFO NoChanges: 1
08:24:36 BIOCONDA INFO UpdateInProgress: 1
08:24:36 BIOCONDA INFO SUM: 4
08:24:36 BIOCONDA WARNING Switching back to master

It still didn't seem to succeed in the autobump, but at least it didn't crash. It's surprising that the default command in the guideline crashed when shuffling, so I thought I should report this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@niemasd