Make Linux dynamic linkage a first class citizen

[dweckmann]

---

Is your feature request related to a problem? Please describe.
Currently, and on the contrary to Windows platform, the official triplet on Linux is the one with static linkage. There are many problems with that:

1. The CI only test static linkage, which means the quality is not always paired with static build.
2. Static linking make linking big libraries like Qt very slow. Really, when debugging, this is painful
3. License problem: LGPL forbid static linking. Using static linking means your code will be GPL licensed. This is not acceptable for many of us.

Proposed solution

• Add an official dynamic linkage triplet for linux, make them default, like on windows
• Add a CI step to test it

[JackBoosY]

Hmm... @BillyONeal what do you think about?

[BillyONeal]

We choose static linking on the POSIX-ish platforms because dynamic linking generally forces you to use the system-provided libraries (e.g. those acquired through apt). It doesn't matter if you want to use the vcpkg version of boost in general, for example, because all the operating system's default load library paths are going to look in places owned by apt or dnf or similar.

Windows is different because on Windows your app can use version 1 of a shared library, and someone else can use version 2 of that shared library, and as long as those versions have different DLL names, a program can use both versions, since all symbols from that library are namespaced into the DLL in question. In contrast, in POSIX world, all the names go into the shared process symbol table, so trying to load multiple versions of a library into the same process will usually result in memory corruption.

(This is part of the problem domain and affects all non-system-wide package management solutions, so our ability to influence this is limited)

[dweckmann]

Yes, you are indeed right.

However, checking, at least, that the package build correctly on the CI would still increase the quality of the dynamic linkage packages. I guess the package must build out-of-the-box, in all case, even if static linking is the default (which I can live with it).

In the posix world, you have some mean to control where the dynamic loader finds their libraries (most notably by setting LD_LIBRARY_PATH to VCPKG 'installed' lib path). Another option, but this is harder, would be to use rpath at linking stage to ensure to load the dependencies of a library with a relative path from themself, so not the system one...

Agree, this is not easy, but other package management (Conan for stance) are able to (or they claim to) somewhat ensure that.

[BillyONeal]

Agree, this is not easy, but other package management (Conan for stance)

While it's true that other package managers have decided to make dynamic linking on the POSIX platforms a more first class citizen, we believe that leads customer confusion; hence our strong recommendation of static linking on those platforms, and that triplet's community status.

We do have x64-linux-dynamic, and we do accept PRs that improve the experience for that triplet. However, even for folks who only use vcpkg locally, we believe it is too easy to accidentally get runtime libraries from the system at runtime resulting in "impossible" memory corruption, and we don't want to handle that support burden right now. For anything targeting wider distribution, those issues are much worse, since it's it might work due to rpath or similar on the dev box and then break when copied to an end user.

Despite the above, if some work is contributed to improve the experience for dynamic on those platforms, such as copying the right .sos in integrated build systems on the POSIX-ish platforms, that might be a reason to take that triplet out of community status. I think changing the default will remain off the table.

[dweckmann]

Yes, we already know the implication of this linkage type. However, it's not a matter of taste for us. The licence and the GPL contamination is a no go, as, even if most of our work are free and open (take a look at https://github.com/IRCAD-IHU/sight), some external part, are proprietary (just because it uses confidencial hardware, software, stuff that waits for a publication in a medical journal, ..) and we have no way to change that. Using for example static Qt (or any LGPL libraries) would mean we cannot go further than private and confidential prototype. No way to use our software in a OP room.

If your strong recommendations stays on static linkage, that's fair and understandable. We only beg for a CI support, to be sure that things compiles and that a submission doesn't break everything. We are in the process to evaluate VCPKG as a replacement for conan, and that is a real concern for us.

Anyway, thanks for listening :)

[BillyONeal]

We do have x64-linux-dynamic,

Correction, we don't have that 😳

[Osyotr]

UPD: x64-linux-dynamic is now included in vcpkg! #23035

I've been able to build some ports with this overlay triplet:

set(VCPKG_TARGET_ARCHITECTURE x64)
set(VCPKG_CRT_LINKAGE dynamic)
set(VCPKG_LIBRARY_LINKAGE dynamic)

set(VCPKG_CMAKE_SYSTEM_NAME Linux)

# set rpath for libraries
if(PORT MATCHES "fribidi|harfbuzz|pkgconf")
    # Meson, nuff said
    set(pretty_ORIGIN_name "\$ORIGIN")
    set(VCPKG_LINKER_FLAGS "-Wl,-rpath,${pretty_ORIGIN_name}:${pretty_ORIGIN_name}/../lib")
elseif(PORT MATCHES "python2")
    # setting rpath here breaks build and I have no idea how to fix it.
    # But since python2 has no dependencies, not setting rpath is ok
else()
    set(VCPKG_LINKER_FLAGS "-Wl,-rpath,'$ORIGIN':'$ORIGIN/../lib'")
endif()

[MrBolt2005]

Why the specific treatment to packages such as fribidi, harfbuzz and pkgconf? I didn't get it yet, really...

[Osyotr]

@MrBolt2005 because they use meson, which is an "extraordinary
special kind of flower". There is a bug somewhere when you pass strings with quotes and I really don't want to dig in it.

[abique]

@BillyONeal I think there are new options today with dlmopen and RTLD_DEEPBIND. But I'm not a shared object expert.

[JackBoosY]

To add dynamic UNIX support, we must first deal with the default behavior of the system to find dynamic libraries.

[Neumann-A]

add a new toolchain (like https://github.com/microsoft/vcpkg/blob/master/scripts/toolchains/linux.cmake) call it linux-dynamic.cmake (or do magic flags via inspection of VCPKG_LIBRARY_LINKAGE as the windows toolchain does) and add the linker flag there.
Maybe decide on a better location for toolchains then script/toolchains, e.g. triplet/toolchains (since the toolchain is really part of the triplet) and set(VCPKG_CHAINLOAD_TOOLCHAIN_FILE "${CMAKE_CURRENT_LIST_DIR}/toolchains/linux-dynamic.cmake")

[dg0yt]

I would prefer an rpath of $ORIGIN/../lib, and post-build adjustments as needed for vcpkg's unusual file system layout for executables. Or the combination $ORIGIN/../lib:$ORIGIN which would allow for traditional UNIX layouts and for Windows-style side-by-side deployment.

[Osyotr]

@dg0yt this would also fix ports that don't copy tools dependencies.

[Neumann-A]

I would prefer an rpath of $ORIGIN/../lib, and post-build adjustments as needed for vcpkg's unusual file system layout for executables. Or the combination $ORIGIN/../lib:$ORIGIN which would allow for traditional UNIX layouts and for Windows-style side-by-side deployment.

I would argue that this is a custom triplet/toolchain.

[dg0yt]

this would also fix ports that don't copy tools dependencies.

@Osyotr It is not so easy. vcpkg doesn't install tools to bin or tools but to tools/${PORT} and tools/${PORT}/bin, plus tools/${PORT}/debug/bin. Cf. #17607.
If there was some interest in cleaning up the installation layout for tools, it would be possible to use $ORIGIN/../../debug/lib:$ORIGIN for the debug variant. (Copying tool dependencies is mandatory for windows anyway. However, it would be attractive to omit copying on platforms that have rpath.)

[MrBolt2005]

Hummm...🤔 Enviado do Email<https://go.microsoft.com/fwlink/?LinkId=550986> para Windows De: ***@***.***> Enviado:domingo, 29 de agosto de 2021 13:07 Para: ***@***.***> ***@***.***>; ***@***.***> Assunto: Re: [microsoft/vcpkg] Make Linux dynamic linkage a first class citizen (#19127) @MrBolt2005<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMrBolt2005&data=04%7C01%7C%7C8c5e030d293541c75d1708d96b0727ba%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637658500742941343%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Er64CIE1SrW7LYcFAT7AA7PySR9g4bC17wA8RCjb8ls%3D&reserved=0> because they use meson, which is an "extraordinary special kind of flower". There is a bug somewhere when you pass strings with quotes and I really don't want to dig in it.

[davidchisnall]

As the POC shows, you can use -rpath to define the shared library location. Note that this can be relative to $ORIGIN, which is resolved at load time to be the path of the binary, so you can do exactly the same thing that you do on Windows: install a binary into a directory and install its libraries along with it. It's generally considered bad form to put libraries in a bin directory on *NIX platforms and so the common solution for software wanting to use this model is to set rpath to something like $ORIGIN/../lib/myprogname/ and install shared libraries in $PREFIX/lib/myprogname.

One other possibility would be to install shared libraries as {libname}-{build hash}.so and expose that as the library name. This would allow many different versions and configurations of shared libraries to coexist on the same system.

With respect to the original motivation, note that the LGPL does not 'forbid static linking.' and it is categorically not true that 'Using static linking means your code will be GPL licensed.' I am not a lawyer and neither is the original poster, please talk to a lawyer if you need advice about software licensing.

[dweckmann]

You are indeed right, I should have said "If you use static linking, you must release your code as LGPL or provide a way to relink your code with another version of the LGPL library". It is however not so easy to provide a way to relink statically linked commercial proprietary binaries.

Guide for beginner:
https://copyleft.org/guide/

[Cazadorro]

We are also running into this issue. We primarily develop on linux, and while we appreciate the initial decision to mainly use static linking and agree with the reasoning behind it, LGPL and GPL really screw us over on Linux, is there not even a workaround for these issues? We've had to basically halt all VCPKG work in our group because of this, and try to shift gears to Conan because this is such a big legal issue for us.

[BillyONeal]

Does conan actually fix this issue? The issue is still pending discussion because the behavior of the OS loader is hostile to a good experience here, not because the vcpkg maintainers don't care.

[Osyotr]

UPD: x64-linux-dynamic is now included in vcpkg! #23035
I've built my app with patches from first PR and it looks promising. No LD_LIBRARY_PATH required, even Qt did successfully found its plugins. The only issue I didn't fix yet is rpath in debug tools but who use them anyway :)? Second PR is also relevant, but I didn't tried it yet.
#23035
#19940

[dweckmann]

Well our experience with conan was even worse, but things may have changed (we tried it two years ago). The dynamic linking is well supported (really...), BUT, the main issue we had is that not everything are glued together. It's a collection of recipes, some may work with others, some simply not because the version doesn't match, patches break compatibility with third party libraries... Having a project with a lot of dependencies, with custom build option like cuda/cudnn, with openCV, Qt, VTK, was just very hard to maintain. As I said, maybe things are in a better shape now..

Anyway, for us, it was finally too complicated and time consuming, so we decided to stick to default system libraries and to stick to a particular Linux distribution (the last Ubuntu), changing libraries when upgrading the distribution. Retrospectively, the sanest choice we did.

We really would love to have a working dynamic VCPKG on Linux, it would allow us to share the same version / build system, and to use it on any linux distribution easily. The dynamic shared library support is is really the thing that have stopped us to use VCPKG on Linux.

My two cents...

[Cazadorro]

@BillyONeal Conan works as you'd expect, with the problems you'd expect AFAIK, you use RPATH. We basically don't want to use the system libraries ever on linux, because we run into so many outdated libs from the system repo, so same process dynlib names causing issues shouldn't be a problem for us. Forcing RPATH works for us.

• If we didn't have to worry about these licensing issues, we'd be 100% happy with the solution presented by VCPKG. Licensing really is the sticking point here.
• We still want static linking for virtually everything else that isn't LGPL. We don't need a general purpose solution that works for every library.

We also are just looking for a work around, that would be enough for us.

[JackBoosY]

The most suitable solution needs to be discussed in depth.

[RealLitb]

I haven't yet understood what exactly the problem is. Is it that Linux also searches its system provided libraries? Link the programs with -z nodeflib. Is it that Linux finds system provided libraries first? Link with -rpath and make use of $ORIGIN.

Is it that Linux shared libraries have a single global namespace for all globally visible symbols? That's by design. Why would that be a problem if you take the first two measures?

I'm not experienced with VCPKG, but why do you want to forbid loading of system libraries? If VCPKG is building a self-contained system, then it sounds to me the correct choice would be to execute the binaries in its own container/chroot instead.

[BillyONeal]

If two SO files are incompatible, they should not get the same SOname

Also this is no help because the SOname is gone once you get to process symbol table time. Only one boost::Xxx gets picked.

[RealLitb]

Maybe VCPKG should add a "vcpkg" into the SONAME of each library and use an ldconfig pass afterwards to create symlinks named according to the SO-name (the non-link on-disk name doesn't need to be different).

That would cause basically all dependency finding to fail, since build systems aren't going to be looking for our funny renamed copies.

The trick is that the copy is not renamed. The file would still be called libboost.so.1.2.3 but if you run ldconfig on it, a symlink liboost-vcpkg.so.1.2.3 is created and points to it. At least that was my trick-plan. Not sure if it would work out. But I didn't think about the runtime symbol table. As you said correctly, only one symbol of them gets picked.

[Osyotr]

FYI, x64-linux-dynamic is usable now. IIRC, the reason why it is not tested in CI is because there's no budget for it.

[RealLitb]

I tested my trick but to prevent symbol collision at runtime, I also passed --default-symver, which automatically adds the SOname to the dynamic symbol name. I have no idea what packages this approach would break. Given this source:

#if !MAIN
void shouldbeversioned() { }
#else
void shouldbeversioned();
int main() {
    shouldbeversioned();
}
#endif

And this command line

g++ -std=c++20 -O2 -Wall -pedantic -pthread -shared -fPIC -Wl,--default-symver -Wl,-soname libaout-vcpkg.so -o libaout.so main.cpp && readelf -W --dyn-syms -d libaout.so && g++ -std=c++20 -L. -laout -O2 -Wall -pedantic -pthread -DMAIN=1 main.cpp && ln -s libaout.so libaout-vcpkg.so && readelf -W --dyn-syms -d a.out && LD_LIBRARY_PATH=. ./a.out && echo success

It outputs

Dynamic section at offset 0x680 contains 28 entries:

  Tag        Type                         Name/Value

 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]

 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]

 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]

 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]

 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

 0x000000000000000e (SONAME)             Library soname: [libaout-vcpkg.so]

....


Symbol table '.dynsym' contains 13 entries:

   Num:    Value          Size Type    Bind   Vis      Ndx Name

....
     6: 00000000000005e0     1 FUNC    GLOBAL DEFAULT   11 _Z17shouldbeversionedv@@libaout-vcpkg.so
...

Dynamic section at offset 0x7b8 contains 29 entries:

  Tag        Type                         Name/Value

 0x0000000000000001 (NEEDED)             Shared library: [libaout-vcpkg.so]

 0x0000000000000001 (NEEDED)             Shared library: [libstdc++.so.6]

 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]

 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]

 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]

 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
.....

Symbol table '.dynsym' contains 8 entries:

   Num:    Value          Size Type    Bind   Vis      Ndx Name

     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND 

     1: 0000000000400674     0 FUNC    GLOBAL DEFAULT   14 _fini

     2: 00000000004004b0     0 FUNC    GLOBAL DEFAULT   10 _init

     3: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND [email protected] (2)

     4: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_deregisterTMCloneTable

     5: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __libc_start_main@GLIBC_2.2.5 (3)

     6: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __gmon_start__

     7: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_registerTMCloneTable

success

It demonstrates that you can link to -laout (because there's a libaout.so, which is usually a (-dev) symlink aswell) but then at runtime the aout-vcpkg is loaded and also the symbols are versioned with the aout-vcpkg in it. I'm not experienced with versioning, but maybe this or a variant of it could give a rock-solid experience.

[RealLitb]

FYI, x64-linux-dynamic is usable now. IIRC, the reason why it is not tested in CI is because there's no budget for it.

Like @BillyONeal said (and I think he has a good point), the major reason seems to be that system-SO files will correctly load a required system lib from the system but as symbols are same-named, the dynamic symbol resolution resolves symbols in them in favour of VCPKG's versions (they are first in breath-first order). This issue is not fixed by -rpath.

[VioletGiraffe]

There is now x64-linux-dynamic, thank you for that, but why no arm64-linux-dynamic?

[MaxRev-Dev]

@BillyONeal @dg0yt What if the rpath gets overridden? Could you please give an advice here?

Consider a .NET app which has libssl.so.3 dependency running on Debian 12 (SSL 3 breaking change). This libssl.so.3 has version 3.0.2. Docs on how dotnet loads native libraries are here.

So, I'm building curl with openssl dependency of version 3.2.0.
On start the app loads the system ibssl.so.3 (3.0.2). Then it fails on curl.

/lib/x86_64-linux-gnu/libssl.so.3: version `OPENSSL_3.2.0' not found (required by /app/bin/Debug/net8.0/runtimes/linux-x64/native/libcurl.so.4)

[dg0yt]

The short question probably isn't a good answer...

1. Your final application needs to be made aware of the runtim lib locations, too. Did you set it up at all?
2. Basically you shouldn't mix different binaries of the same library. If you are forced to use the system openssl library, you can use an empty overlay ports for openssl to make all binaries in vcpkg use that system library. But this quickly becomes an advanced task.

[MaxRev-Dev]

@dg0yt This one is already a bit complicated, but anyways..

1. It looks like I'm forced to use SSL loaded by runtime. I'm compiling packages for a library. The runpath/rpath is set to the following.
   0x000000000000001d (RUNPATH) Library runpath: [$ORIGIN:$ORIGIN/runtimes/linux-arm64/native]
   I tried to set LD_LIBRARY_PATH to point to the executable directory for a single run and it works. However, it's not a workaround at all as I can't include that into the library without an additional executable wrapper.
   But regarding loader paths I'm getting the following LD_DEBUG output:

Details

     15675:
     15675:     calling init: /usr/share/dotnet/shared/Microsoft.NETCore.App/8.0.0/libSystem.Security.Cryptography.Native.OpenSsl.so
     15675:
     15675:     find library=libssl.so.3 [0]; searching
     15675:      search cache=/etc/ld.so.cache
     15675:       trying file=/lib/aarch64-linux-gnu/libssl.so.3
     15675:
     15675:     find library=libcrypto.so.3 [0]; searching
     15675:      search cache=/etc/ld.so.cache
     15675:       trying file=/lib/aarch64-linux-gnu/libcrypto.so.3
     15675:
     15675:
     15675:     calling init: /lib/aarch64-linux-gnu/libcrypto.so.3
     15675:
     15675:
     15675:     calling init: /lib/aarch64-linux-gnu/libssl.so.3
     15675:
     15675:     /lib/aarch64-linux-gnu/libssl.so.3: error: symbol lookup error: undefined symbol: SSL_state (fatal)
     15675:     /lib/aarch64-linux-gnu/libssl.so.3: error: symbol lookup error: undefined symbol: ERR_put_error (fatal)
     15675:     find library=libc.so [0]; searching
     15675:      search cache=/etc/ld.so.cache
     15675:      search path=/lib/aarch64-linux-gnu:/usr/lib/aarch64-linux-gnu:/lib:/usr/lib            (system search path)

2. It should be possible to inject an empty overlay into the build pipeline as I have several dependencies on OpenSSL.
   However, I'm building on Debian 11 as it has the lowest widely used version of glibc. It has libssl1.1 from the system packages. I moved from CentOS as it was difficult to maintain all dependencies. Then I started using Debian 11 until last year when the dotnet team moved to Debian 12. I'm not sure to which version stick to.

I saw this comment by @BillyONeal on linking statically, but that would increase package size. And for now, it's not an option because GDAL fails to link to a static curl library. I'd consider it as a last resort choice.

For now, I tried to specify the major/minor versions and it works without any issues.
Can I include that as a post-build step but would that introduce any issues?
And the question remains - is it possible to extend naming to include specific versions for shared libraries with vcpkg?

patchelf --replace-needed libssl.so.3 libssl.so.3.2.0 libcurl.so.4
patchelf --replace-needed libcrypto.so.3 libcrypto.so.3.2.0 libssl.so.3
mv libcrypto.so.3 libcrypto.so.3.2.0
mv libssl.so.3 libssl.so.3.2.0

[dg0yt]

And for now, it's not an option because GDAL fails to link to a static curl library.

Not a known issue. But with curl, we quickly get to openssl again.

I can't answer the other questions for this context.

[MaxRev-Dev]

@dg0yt thank you

[MaxRev-Dev]

Regarding SSL version compatibility, it looks like I can do this patch by specifying the minor/patch version.
Could anyone from the VCPKG team advise on how to add the patch version to the package?
Thank you

[tsondergaard]

Not having dynamic linking supported as a first-class citizen on Linux is a problem for my company's project as well. One of our products alone consists of 30+ executables. Many of them use Qt libraries which are large libraries so this would make our installation many times larger. We also use Qt Webengine which cannot be built as a static library. Finally we use Qt under the LGPL license, which makes static linking a problem.

@BillyONeal wrote:

We choose static linking on the POSIX-ish platforms because dynamic linking generally forces you to use the system-provided libraries (e.g. those acquired through apt). It doesn't matter if you want to use the vcpkg version of boost in general, for example, because all the operating system's default load library paths are going to look in places owned by apt or dnf or similar.

With the proper RUNPATH set this is a manageable problem.

Yes, there are caveats and dragons with dynamic linking but as I have pointed out above it is simply required for a lot of projects.