Announcements

[dpryan79]

This issue will serve as another channel for making project-wide announcements. Please avoid discussion in this issue, post it in the gitter channel instead.

[dpryan79]

22.02.2020: There was an issue with markupsafe incompatibility with the jinja2 dependency used by bioconda-utils. As a consequence all builds were failing. To resolve that we have made a new bioconda-utils release. We were already in the process of performing a libdeflate migration in the bulk branch, so now all PRs and the master branch are also using that new pinning.

Please note that we are also using GCC 10 (like conda-forge)! This is causing some build failures in the bulk branch, but hopefully they can all be resolved relatively easily.

[dpryan79]

The commands "@BiocondaBot please merge" and "@BiocondaBot please fetch artifacts" should be working again.

[johanneskoester]

Dear all. I just started a big update of our dependency pinnings. This will lead to using the same pinnings as conda-forge again, including e.g. openssl 3 (see here for details). We have further decided to also follow conda-forges current Python pinnings. This means that we drop support for Python 2.7, as well as Python 3.6 and 3.7. This does not mean that Python 2.7 packages do not work anymore in Bioconda. However, you have to provide your own conda_build_config.yaml next to the meta.yaml in such a case, which then has to add Python 2.7 as a version to build for.

As always the pinning update first happens within our special bulk branch. Once we have updated all existing packages, the bulk branch will be merged into the master branch and github.com/bioconda/bioconda-common will be updated to also use the new pinnings. Subsequently, new or updated packages will be automatically built with the new pinnings.

[dpryan79]

As noted in gitter and #41025 CI testing on Azure is currently down. That means that no PRs can be tested or merged at the moment. We've contacted Microsoft but don't expect this to be resolved for another 1-2 days.

[dpryan79]

We have migrated everything to github actions and packages are building again. Please note that (1) artifacts are still not being stored, (2) the bot will have to be reprogrammed once they are and (3) the master branch currently needs to rebuild everything prior to uploading. Those tasks will be worked on in that order.

[corneliusroemer]

For those wondering about Python 3.11 (and Python 3.12) support, this is on the core team's mind but their priority is currently on getting ARM builds ready, and a bioconductor release. 3.11/12 should get attention from January 2024 onwards.

See this comment from @bgruening:

Yes, we intend to support Python 3.11 and 3.12. However, we are currently busy rolling out the ARM builds and in parallel the BioConductor release. This keeps us and the CI already busy. Realistically we can only care about 3.11/12 in January. Sorry.
(from bioconda/bioconda-utils#938 (comment))

[corneliusroemer]

@daler gave an (unofficial) update on current plans re ARM and Python 3.11/3.12 in a bioconda-utils issue. I'm copying it over here for better visibility:

Here's a snapshot of the current state and short-term goals:

Current goal is to get everything cleaned up and in shape to support large-scale updating for ARM and Python 3.11 specifically regarding the build-failure yamls and their interaction with the bulk and master branches. For example, right now removing a build-failure yaml is insufficient to trigger a rebuild; we need to work out the right process for having many people work on build failures simultaneously. This requires the technical component (actually writing the infrastructure code) but also the sociological component (how, specifically, do we best coordinate across contributors without stepping on each others' toes)? This is important for both ARM and Python 3.11 and of course any future migrations/updates. The “practice run” for this, once the infra is in place, will be finalizing the remaining Bioconductor packages.

There are a last few remaining details to be taken care of with ARM containers to avoid arm64 clobbering amd6. This involves quay.io, docker manifests, and making sure all the infrastructure is updated to handle all of that. Then we’ll kick off some opt-in ARM builds to make sure everything works in practice.

Then it’s on to Python 3.11. Hopefully the infrastructure will be in place there for rapid building on bulk, kicking out build failures to the wiki, and coordinating with the community to fix everything. My dream is for the 3.11 to go quickly and smoothly. Ideally, 3.12 could follow shortly after. And by then hopefully we'll have worked out the kinks on ARM, and can scale up those builds as well.

The challenging part in all of this is that to make meaningful progress on infrastructure components like ARM containers and the build-failure process, we need to load everything into mental RAM. That is difficult to do when we all have day jobs, many of us are trying run our respective labs and working on this in our (increasingly rare) spare time. So then documentation becomes a vital part of all of this to minimize the mental-RAM-loading time (e.g. https://github.com/bioconda/bioconda-docs/pull/16/files). The docs are certainly helping. Ideally, we would document everything extensively enough so the community can jump in and help out and alleviate the bandwidth issue. We’re not there yet,but we're getting there.

We of course also need to be better about the decision-making process, planning process, and documenting all of the various processes and moving parts to make this all more transparent to the community. Developing that documentation and communication takes away from making progress on working on infrastructure, so it's a balance.

None of this helps directly with Python 3.11, I know. Just trying to give some insight into how we're trying to improve all the internal processes and make it better for everyone over the long term. Rest assured that 3.11 is definitely in the works.

copied from bioconda/bioconda-utils#938 (comment)

[daler]

You can now build linux-aarch64 packages. 🎉

See https://bioconda.github.io/developer/aarch64.html for details on how to do this. Thanks for everyone's patience on this.

For now, this is opt-in. Once things are working smoothly, we'll switch to building by default.

We still don't have linux-aarch64 containers building. It is proving tricky to orchestrate containers built across multiple CI platforms to put them into a single OCI manifest. But we have some ideas, and once we have something implemented we will build containers for anything missing (i.e., for any packages built between now and when we have that implemented).

Documentation in general has also been updated. The goal here is to better explain all of the moving parts and encourage more involvement from the community on the infrastructure. Specifically:

• the aforementioned page for aarch64
• "date changed" indicators so you have a better idea how new something is
• a new repository inventory
• a new dockerfile inventory, with explanation on how the build-env, create-env, and base images all work together
• a new CI inventory to help keep track of what is happening and where
• platform nomenclature for keeping track of the sometimes-confusing names
• fixed package download plots on the front page
• consolidated the 2 faqs into one
• removed the conda-build-3 page

[bgruening]

Thanks a lot, @daler! Awesome work!

[daler]

Some updates:

1. linux-aarch64 builds are still progressing as opt-in, as described in the above comment. Some packages work with no problem, but there are also multiple packages with disparate issues that the community is working out. Some base packages (dependencies of many other packages) are still being worked on, which blocks many other packages. But there has been a lot of progress.

2. Rebuilding all Python packages for Python 3.11 is planned for the near future. This needs ~2 wks of volunteer time to manage, and will be starting soon. During this, we will be testing functionality for better democratizing the migration process so it needs less single-person time commitment in the future.

3. We are starting to look into building on macOS ARM support for bioconda-utils using GitHub Actions. No idea how much effort this will be, what the limiting step will be, or if the overhead of building in bioconda-utils support for yet another CI platform is worth it (vs waiting for CircleCI to support for public repos). So there's no timeline, just the strong intention to support it.

If you want to help then a great thing to do is to get familiar with the build system(s) so we have more people available to contribute. This would mean reading through the bioconda-utils code and the following bioconda documentation sections (along with all code points-of-interested linked therein) to get familiar with the build systems:

• repository inventory
• dockerfile inventory
• CI inventory

While going through those resources it would be amazing if you could add/update documentation for bioconda-docs to speed the learning process of the next person.

[aliciaaevans]

The latest bioconda-utils version (3.3.0) will only be used on the Bulk branch for now as we start the bulk updates for Python 3.11 and 3.12. It will take a while to work through building everything, even assuming there are no major issues, so I will plan to provide some updates here as things progress.