[feature] Dependabot like tool to automate updating lockfiles

[stackfun]

What is your suggestion?

Unfortunately, dependabot has stopped accepting feature requests to support other package managers. See this github thread here

A tool like this would greatly supplement Conan lockfiles.

For our internal components, we're going to ask teams to start checking in the lockfile beside their conanfile. However, teams are hesitant to adopt this as it is another process that they need to maintain, and it's not something they're not familiar with. If this process were automated, I'm sure there would be greater adoption.

Here are a few key features in dependabot that would be useful

• automatically creating a PR with an updated lockfile
• ignore updating subset of dependencies

Conan specific features/considerations

• Invoke conan config install
• Find the latest version which matches the version ranges from conanfile
• Support checking dependencies for mutliple configurations (windows/linux dependencies can be different), and save the dependencies in the same lockfile
• Make sure updates still resolve to a viable conan graph (no conflicts with other dependencies)

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[samuel-emrys]

There has been some work to automate the process of bumping dependencies in the conan-extensions repo. Specifically: https://github.com/conan-io/conan-extensions/tree/main/extensions/commands/recipe.

These extensions are community maintained but centralized. There currently isn't anything for lockfiles, but I can see the utility you're talking about and I think additional logic would be welcome there.

I realise your request is for a bot, but this could be a good first step.

[stackfun]

Or additional features related to lockfiles which will make developing such a bot easier.

For example, npm outdated which does a bit more than the extension you shared.

I'm personally looking for solutions that will also work in a private conan repository.

[memsharded]

Hi @stackfun

Thanks for the suggestion
This wouldn't be a Conan client feature, in any case, maybe it is something to be considered as an extension. So I am moving this to the conan-extensions repo.

Also, at this moment, it doesn't sound like something we could prioritize enough, so it will probably rely on the community, or need to wait some time. Thanks!

[SvenStaehs]

there's been an update, dependabot is now accepting community contributions for new ecosystems again: dependabot/dependabot-core#1616 (comment)

It does still state that they want "an ecosystem maintainer [...] interested in integrating with Dependabot, and [...] willing to help provide the expertise necessary to build and support it", so ideally, someone from jfrog/conan team, not just some random user with some half-knowledge about how it might be supposed to work.

If this doesn't apply to anyone here, any idea whom best to contact? Should people lodge support tickets with jfrog directly? Especially enterprise customers might have some traction that way...

[memsharded]

Thanks for the heads up @SvenStaehs

At the moment we have a long roadmap, too many high priority things on the backlog, so it seems we cannot put resources from the team in the very short term. But lets have a look first just in case and think a bit about it.

Just a couple of quick notes:

• It is not possible/enough to update a lockfile. Updating a lockfile doesn't work if the conanfile is not updated.
• conanfile.txt might be easier to update by conanfile.py cannot be automatically updated, as it is python code that can contain conditionals and other logic, so not possible to modify it.
• So the automatic update might be a bit challenging. Informing about possible updates in dependencies, that sounds more doable.
• Conan will probably be getting a new conan graph outdated command that will report over new versions of dependencies (not exactly the same, but very related, could help)

[stackfun]

We ended up creating an internal tool for this. Our bot requires a conanfile and the list of profiles the team builds against. On a configurable shcedule, it creates the full dependency graph for each profile, dedupes the resolved recipe revisions, and stores/updates the lockfile, and submits a PR with a description about which dependencies changed.

There's lots of functionality that's missing compared to Github's dependabot, but still has proven to be extremely useful. Unfortunately I probably can't share the tool anytime soon as it uses our own wrapper Conan API's to add custom functionality.

[SvenStaehs]

Thank you @memsharded for the assessment! And to @stackfun for your take.
We may end up imposing restrictions on our conanfile.py files (preventing complicated logic) in order to allow us to write a custom tool that does update them. I already wrote one that just makes a string replacement, but I'm not happy with it yet.

Obviously any change thus created needs to go through PR review before it can be merged, so it's not a show-stopper if it's not perfect, but it's understandable that it's not the highest priority for conan team to spend effort and end up with a fundamentally flawed tool.

[memsharded]

We may end up imposing restrictions on our conanfile.py files (preventing complicated logic) in order to allow us to write a custom tool that does update them. I already wrote one that just makes a string replacement, but I'm not happy with it yet.

If you plan to impose restrictions, maybe using the conandata.yml is the best approach. Don't let recipes define directly requirements, but load from self.conan_data, and a bot can more easily propose changes to the yaml file.

Thanks for the feedback!

[stackfun]

If your conanfile had version ranges like pkg/[^2.0.0], then the approach to update the lockfile instead of the conanfile may work too. In this way, updating the major version will not be managed by the bot which is a behavior you may be interested in.

In addition in our setup, each of our jenkins build creates a new recipe revision, so the lockfile helps control the builds of each internal component. In this way, our bot is still useful even without version ranges in the conanfiles.