Should molecule v5 remove testinfra verifier?

[ssbarnea]

As part of the upcoming refactoring molecule v5, we would want to remove support for custom verifiers and keep only the ansible verifier.

Now is the time to speak if this change affects you in a negative way, so we can see what can be done. The maintenance of the project made it hard for the project to evolve, so removing less essential parts out of it is key for allowing it to move faster.

If you use another verifier I would also be a good idea to post a comment so we can ask for more information.

PS. No decision was made!

[apatard]

What's the exact plan here ? Remove all verifiers and hardcode things to allow only the ansible verifier ? Why ?
If the problem is only that having the verifier inside molecule is annoying, why can't it be put inside its own repository (or create a repository with all "external" verifiers) and handled the same way we're doing for drivers ?

[ssbarnea]

We are trying to slim down molecule to reduce its code complexity in order to make it maintainable and easy to use. One of the issues the project had is that it grew in too many directions and created too many ways in which it can break.

If we fail to reduce the scope, we might endup having to come up with an simple alternative that uses the same basic principles, like writing playbooks to test ansible, scenarios with steps.

Taking testinfra out is it an option but it is likely to sorts only ½ of the the issue.

@apatard Regarding driver separation, it opened a different set of problems as their maintenance become much harder as required repeating operations on ~15 repos instead of one. It is true that some drivers deserve to die, but it is very hard to identify which ones worth being saved or not. That makes the molecule ecosystem maintenance bit more challenging.

One of the things I am investigating is a model where we no longer need drivers as python packages and that we can make an ansible collection also act as a driver. For example a podman collection could have the extra metadata needed for allowing to use it with molecule.

A second thing that I am considering is replacing the molecule.yml config file with a simple vars file that is loaded by ansible. If everything goes well we might be able to perform the testing by calling ansible directly, molecule becoming only a thin wrapper that makes it easier to run these tests, with shorter command line than ansible and with ability to log and report the test execution.

PS. Regarding surveying the popularity of various drivers, I just realised that I could get some data by performing some searches on https://sourcegraph.com/search

[apatard]

We are trying to slim down molecule to reduce its code complexity in order to make it maintainable and easy to use. One of the issues the project had is that it grew in too many directions and created too many ways in which it can break.

If we fail to reduce the scope, we might endup having to come up with an simple alternative that uses the same basic principles, like writing playbooks to test ansible, scenarios with steps.

Taking testinfra out is it an option but it is likely to sorts only ½ of the the issue.

fwiw, my limited minds prefers testing things with python and testinfra. Seems easier for me, even if I don't have examples at hand.
Moreover, rewriting/migrating tests to use ansible verifier seems painful from a user point of view.

@apatard Regarding driver separation, it opened a different set of problems as their maintenance become much harder as required repeating operations on ~15 repos instead of one. It is true that some drivers deserve to die, but it is very hard to identify which ones worth being saved or not. That makes the molecule ecosystem maintenance bit more challenging.

that's why I talked about having one repo for all removed verifiers. I'm possibly wrong on that but the CI should allow us detecting "dead" verifiers.

One of the things I am investigating is a model where we no longer need drivers as python packages and that we can make an ansible collection also act as a driver. For example a podman collection could have the extra metadata needed for allowing to use it with molecule.

Seems doable but I'm not sure what it will provide. For instance, you're assuming that the collection will have proper CI and maintenance to catch issues and move the maintenance load there. I'm not convinced it'll work in practice. Just remember the discussion here: ansible-community/molecule-podman#142

A second thing that I am considering is replacing the molecule.yml config file with a simple vars file that is loaded by ansible. If everything goes well we might be able to perform the testing by calling ansible directly, molecule becoming only a thin wrapper that makes it easier to run these tests, with shorter command line than ansible and with ability to log and report the test execution.

fyi, doing everything with ansible is more or less what's done by monkeyble (if I understood the tool correctly).

Also, I'm not even thinking about how painful it'll be for users to migrate.

[ssbarnea]

@apatard if you could provide one or two links to relevant repositories making use of testinfra, it could be very useful to understand the differences. Maybe we can find a way to keep it.

And yes, I am very aware of monkeyble and we are watching it closely, thinking on how it fits into the picture. The project has serious value. Basically in January I will start working seriously on thinking about future of non-static testing with Ansible and comparing exactly all the tools, features so we can decide where to invest or cut investments. Who knows, maybe we can find a way to integrate them to make them easier to use.

I very aware of your contributions to the community, so I want to be sure we get this right. Thanks for the feedback.

[jghal]

Clarification on the discussion title, removal of testinfra would be for a v5 release, right?

[ssbarnea]

You are right, I modified the title/body but if I edit the poll, all the votes will be reset.

[davedittrich]

I'll try to formulate a more comprehensive response as I have time, but here are a few quick thoughts.

• testinfra is actually a plugin to pytest (pytest-testinfra) as is pytest-ansible, so isn't the proposal really to eliminate use of pytest as a verifier and only allow Ansible? Is this what you are referring to as "drivers as Python packages?" Many people use pytest for testing their Python projects. Wouldn't fixing the way molecule invokes pytest to take advantage of its broad range of available plugins--beyond just pytest-testinfra and pytest-ansible--serve a larger user base?

• Is this partly driven by problems with pytest-testinfra and pytest-ansible plugins conflicting? After days of debugging and reading source code, I found that pytest-testinfra and pytest-ansible both attempt to add the same Ansible command line arguments at plugin load time, which throws a really opaque exception that molecule can't handle that is very hard to debug due to difficulty in runtime debugging and limitations in logging with molecule. I did come up with a way to resolve the problem but have yet to file PRs to both projects to get them both to work at the same time. TL;DR, you just need to make the arguments plugin-specific. There is apparently an argparse option adding mechanism that pytest has implemented for this (but I haven't found clear documentation on how to use it yet.)

• If ~15 repositories have to be updated to change something, doesn't that suggest and issue with code coupling? That is more a software engineering issue than a reason to limit functionality in a source code project. Improved modularity and code reuse seems like a preferable solution to limiting functionality. I have found a couple of instances where code is cut/pasted across projects and added comments to those repos.

• Wouldn't a generalized "plug-in" style mechanism for invoking verifiers be better in the long run than driver-specific YAML data structures requiring schema validation? A generic mechanism to generate safe command line arguments and use of environment variables passed to child processes should provide enough flexibility.

• In trying to keep molecule testing of my own FOSS project stable, I have noticed regressions and breaking changes that result in making it difficult to maintain stability. Code that worked last month is broken this month. Some of this appears to stem from accepting PRs from humans and code quality bots with insufficient review and testing. Adding runtime integration tests in addition to the static unit tests would help identify regression bugs and other breaking changes before they get merged into what users expect to be stable branches. Adding VSCode launch rules to facilitate runtime debugging would also help produce and review PRs.

[ssbarnea]

When looking from outside is easy to say what maintainers and contributors should do, but the reality is that there are not enough people spending their time to implement these changes. For years molecule was like that and I doubt it will change over the night. That is usually about the time when someone decides to create yet another (slim) project that does the same, or when someone decides to make a fork that than be maintained, again "slim". Basically I was looking for a way to avoid that.

If one has the option to have a project that is supposed to provide 100 features, but most of them are unstable at best, or just 10 but these being well tested and supported, what should be done?

What is the best move given the very restrictive resources that we have? -- by resources I mean active contributors' time.

I love pytest, and I was the one that implemented pytest-molecule 3 years ago. But think how problematic is to use pytest to run molecule which may runs pytest's testinfra, nested!. No wonder that debugging problems is tricky.

Next year I focus on finding a better way to address these. Pytest is still awesome because it provides very useful features like recording output (pytest-html) and producing reports, essential for playbook runs. Still because each coverage test is a chain of subprocesses, its output can be quite long and hard to debug if it appears as a single pytest test.

I am more than curious to hear what other ideas people might have.