How do I use nested vcpkg manifests in submodules/sub-projects?

[mikepricedev]

├── CMakeLists.txt
├── vcpkg.json
├── vcpkg_installed
├── extern
│   ├── CMakeLists.txt
│   ├── sub_project
│   │   ├── CMakeLists.txt
│   │   ├── vcpkg.json
│   │   ├── vcpkg_installed

In the above folder structure the "sub_project" submodule uses vcpkg and a vcpkg.json manifest to manage its deps. The root project also uses vcpkg and a vcpkg.json manifest to manage deps.

However, when the project is configured the sub_project's CMakeLists.txt vcpkg wiring no longer finds packages installed in sub_project/vcpkg_installed dir.
Steps taken:

1. vcpkg install ran in sub_project dir. vcpkg_installed dir was created as expected.
2. PATHS option added to find_package in submodule cmake. (in my case using the frozen lib)

find_package(frozen REQUIRED
  PATHS "${CMAKE_CURRENT_SOURCE_DIR}/vcpkg_installed/${VCPKG_TARGET_TRIPLET}"
)

3. Step 2 finds the library (frozen), but when compiling generates: 'frozen/map.h' file not found #include <frozen/map.h> not an issue in sub_project standalone.
4. The following correctly prints the include dirs.

get_target_property(frozeninc frozen::frozen INTERFACE_INCLUDE_DIRECTORIES)
message(STATUS "frozen INTERFACE_INCLUDE_DIRECTORIES: ${frozeninc}")

Note: using vcpkg as a submodule technique in both i.e.

set(
  CMAKE_TOOLCHAIN_FILE "${CMAKE_CURRENT_SOURCE_DIR}/extern/vcpkg/scripts/buildsystems/vcpkg.cmake"
  CACHE STRING "Vcpkg toolchain file")

[NancyLi1013]

Hi @mikepricedev

Thanks for posting this issue.

You might forget to add target_link_libraries(main PRIVATE frozen::frozen frozen::frozen-headers) in your CMakeLists.txt.

Could you please refer to the usage of frozen to rebuild your project?

 find_package(frozen CONFIG REQUIRED)
 target_link_libraries(main PRIVATE frozen::frozen frozen::frozen-headers)

[NancyLi1013]

@mikepricedev

Please let me know if the above suggestion can work for you.

[mikepricedev]

@NancyLi1013
Thank you so much for taking a look. Before I go further I realized I am making an assumption that I haven't found any specific documentation. Does vcpkg support project's using vcpkg.json that have submodules that also use vcpkg.json? I'm using the following technique to add my submodules to my root project's CMakeLists.txt.

add_subdirectory(extern/sub_project)

In my case, Frozen is a dep of the sub_project managed in the sub_project by a vcpkg.json manifest. When I build the sub_project project independently (not as a submodule) the #include <frozen/map.h> is not an issue.

It is only when the sub_project is built as a submodule of another project using vcpkg. Both projects have vcpkg.json mainfest files. If I hoist the dependencies from the sub_project and paste them into the root project's vcpkg.json dependencies . There is no issue.

I also noticed something else particular. The sub_project also has a dep to "boost-smart-ptr". When I build the sub_project independenly boost is found in the vcpkg_installed dir. When built as a submodule the dep is found from my system's copy of "boost". I do not have a system copy of "frozen".

[PhoebeHui]

@mikepricedev, does this still be a question for you?

From manifests.md, when invoked from any subdirectory of the directory containing vcpkg.json, vcpkg install with no package arguments will install all manifest dependencies into /vcpkg_installed/.

[cyraid]

I'm running into this problem too.. But I'm relying on visual studio 2019 cmake support to use vcpkg. It seems to be ignoring the sub directory/sub directory.

Is there support for add_subdirectory and sub directories with their own vcpkg.json? (And just using the main project without opening separate visual studio 2019 instances with their own cmakelists.txt).

[strega-nil]

@cyraid there is currently no support for having multiple top level manifests. We have considered some ways to solve this, but none of them very good. If you're interested in solving this issue, please open a discussion topic about it. Otherwise, writing a top-level vcpkg.json that contains all of the dependencies your subdirectories need, or alternatively using vcpkg to manage these dependencies as well, are the two solutions we see right now.

[cyraid]

Hi @strega-nil ! Thank you for the reply! Where would you guys like to discuss this? I mean, I don't have much time to talk about it but my quick input would be to add a ADD_TARGET_VCPKG in the sub directory inside that CMakeLists.txt. So, ADD_LIBRARY(MyPlugin), and then ADD_TARGET_VCPKG(MyPlugin). This way you could add the appropriate includes and properties etc. for that target. This would of course also call vcpkg in that directory too, but perhaps with the same cache directory and out from CMakeSettings.json?

[strega-nil]

@cyraid https://github.com/microsoft/vcpkg/discussions

[kentf]

We'd also like to have nested vcpkg.json support.

We have many projects in a small monorepo. Some of the projects require some larger dependencies (OpenCV, Qt), while some only requires header only libraries.

The projects can be added/excluded from the solution by CMake option toggles. It would be nice to only acquire/build, say Qt and OpenCV, if the projects that uses those dependencies are being configured and built.

[ras0219-msft]

You can use features for this: https://github.com/microsoft/vcpkg/blob/master/docs/users/manifests.md#vcpkg_manifest_features

[ras0219-msft]

Vcpkg doesn't support separate vcpkg.json files in this manner; instead, you'll need to add the inner dependencies to the outer vcpkg.json file. For CMake, only the vcpkg.json adjacent to the top-level CMakeLists.txt is considered.

[sketch34]

@ras0219-msft Just so I understand, you're saying currently vcpkg doesn't support vcpkg.json nested in submodules?

In my case the "outer" project does not use vcpkg (yet), but the submodule does. The submodule fails to build as part of the main project, but builds fine on its own.

I set the vcpkg CMAKE_TOOLCHAIN_FILE for the outer project:

        "CMAKE_TOOLCHAIN_FILE": {
          "value": "$env{VCPKG_ROOT}/scripts/buildsystems/vcpkg.cmake",
          "type": "FILEPATH"
        }

The submodule does see the toolchain file correctly but does not find the vcpkg-managed dependency:

1> [CMake] CMake toolchain file: C:/Dev/my_project/submodules/vcpkg/scripts/buildsystems/vcpkg.cmake
1> [CMake] x64 architecture in use
1> [CMake] Building C:/Dev/my_project/submodules/dcn/gfxd as a library
1> [CMake] CMake Error at C:/Dev/my_project/submodules/vcpkg/scripts/buildsystems/vcpkg.cmake:786 (_find_package):
1> [CMake]   Could not find a package configuration file provided by "prometheus-cpp"
1> [CMake]   with any of the following names:
1> [CMake] 
1> [CMake]     prometheus-cppConfig.cmake
1> [CMake]     prometheus-cpp-config.cmake

[alandtse]

Vcpkg doesn't support separate vcpkg.json files in this manner; instead, you'll need to add the inner dependencies to the outer vcpkg.json file. For CMake, only the vcpkg.json adjacent to the top-level CMakeLists.txt is considered.

Is there a design philosophy behind this like to avoid nested dependency chains or some other reason? It'd be nice to have it articulated.

The issue I see is this will limit the practical use of cmake/vcpkg based libraries to those without any of their own cmake/vcpkg based libraries. A downstream user has to manually add to their own vcpkg.json any upstream subdependencies and keep track of any additional subdependencies that may be added by a dependency. That just seems unnecessarily tedious.

[ras0219-msft]

To be crisp about what's unsupported: vcpkg does not support multiple vcpkg.json files consumed by the same CMakeLists.txt source tree. This can be resolved today either by:

1. If there is only one dependency needing vcpkg, simply copy the vcpkg.json from the dependency up to the root CMakeLists.txt and then proceed with the standard vcpkg integration.
2. If there are multiple dependencies needing vcpkg, merge their dependency arrays together into a parent vcpkg.json. This is a simple concatenation assuming they are not using more complicated features like overrides (in which case the user needs to thoughtfully choose which version they actually want to use)
3. Instead of consuming the dependencies as submodules/add_subdirectory(), consume them as vcpkg packages. This will get you binary caching and the full suite of vcpkg dependency management functions. You can use either overlay ports or custom registries to do this.

As for the philosophy behind it: it is not trivial in the general case to merge manifests together. We are not saying it will never be supported, but rather that we have not yet worked through all the design implications.

[sketch34]

This is still an issue, just bumping for visibility and any news.

Specifically, I just encountered this issue again with a top-level non-vcpkg project consuming a submodule that uses vcpkg. It seems like the workarounds are required, although option (3) is not possible in my current case.

[petersteneteg]

At some point I wrote custom cmake "toolchain" file similar to the vcpkg/scripts/buildsystems/vcpkg.cmake one.
It will traverse my current build setup (a fairly complex plugin based with with lots of optional parts) and look for "sub" vcpkg.json files. It will then go ahead and merge those and with the top level one write that in the build dir. Then I set the VCPKG_MANIFEST_DIR to the build dir and call into the regular vcpkg toolchain.

I have some limitation on what can happen in the "sub" vcpkg.json files to make it easier to merge. but generally it seems to work quite well.

[laurentcau]

CMakePresets.json uses an include section to solve this issue.

{
        "version": 4,
	"include": [
		"Public/SDK/CMakePresets.json"
	  ]
}

Could that be an option to solve the issue for vcpkg.json too ?