[vcpkg] support ports unit-tests

[yurybura]

It would be nice to have possibility to write a simple CMake based project to test a port.
This test must run on the CI machines after succeeded port build.

[strega-nil]

This is something we've discussed, but would take a lot of work to implement, and it's something that might make our CI take a lot longer to run. It's going to be a lot of discussion before we can implement this feature.

[BillyONeal]

This isn't something for our regular discussion meeting, dropping the tag.

We could add such a feature without necessarily turning it on in CI.

[JackBoosY]

Please track issue #72.

[strega-nil]

I think that this is a better place for discussion of port unit tests; #72 should be closed as we have CI/CD to make sure that the ports work.

[JackBoosY]

We should only enable test on pipeline test.

[dg0yt]

In #22258, I use a port vcpkg-maintainer-options to enable control over tests.

[mathisloge]

Couldn't we add a vcpkg/ports/<myport>/tests dir which will have a normal CMakeLists.txt setting up one or more minimal example executables and add a vcpkg-tool command test-port <myport> which will configure and build the tests.
Maybe add some cmake presets which configures correct compilers / triplets

[chausner]

Fully agree. I am really suprised that vcpkg does not have any solution in place for this. It would help spotting issues when ports are built correctly but cannot be used properly. I see many bug reports by users that report issues with ports that are due to lack of proper testing of ports. It is not going to get better unless there is some automated way to flag these problems before ports get added/updated in the master branch.

Conan has the "test_package" concept and requires all ports in Conan Center to have a corresponding basic test. This is usually a simple CMake application that links to the library and calls one or a few simple functions to ensure headers are found and the library is correctly linked, as @mathisloge also suggested above. It is not supposed to be an exhaustive test suite or the unit tests of the packaged library. See for example https://github.com/conan-io/conan-center-index/tree/master/recipes/aaf/all/test_package.

I think we can easily require port maintainers to add such a test, it should not be much work and only needs to be written once and usually does not even need to be updated when the library gets updated.

This is something we've discussed, but would take a lot of work to implement, and it's something that might make our CI take a lot longer to run

Compiling and running the test application should only take a small fraction of the time required to build the library itself so I don't think that this would have any significant effect on CI scalability.

[dg0yt]

#22258 and #15173 somewhat go into this direction, by attaching cmake build tests to ports.

From the cmake-user port I can tell that in the MSVC world, CMake config overhead is quite noticable for small libraries if you test multiple configurations (debug, release, different cmake version, different cmake options). It might be mitigated by reusing the same test project at least for similar configurations.