GNU ld --as-needed breaks dynamic loading

[jrw972]

Scenario

Two dynamic libraries create ORBs. The first library SyncClient does not depend on bidirectional GIOP, while the second library OpenDDS_InfoRepoDiscovery does. OpenDDS_InfoRepoDiscovery includes BiDirGIOP.h to satisfy the service configuration framework.

If SyncClient is loaded first, then TAO throws an exception when OpenDDS_InfoRepoDiscovery attempts to use BiDirGIOP

(24541|24541) EXCEPTION, ERROR: InfoRepoDiscovery::add_publication:  user exception, ID 'IDL:omg.org/CORBA/PolicyError:1.0'

If OpenDDS_InfoRepoDiscovery is loaded first, then no exception is thrown.

Investigation

Investigation showed that the exception is legitimate because the BiDirGIOP loader did not register the policy. Hence, the problem is with the loader. Further investigation seemed to show that TAO_BiDirGIOP was being loaded but the loader was never initialized.

Including BiDirGIOP.h in SyncClient solved the problem.

A trail starting here led me to try compiling with LDFLAGS="-Wl,--no-as-needed". This also solved the problem.

I also attempted to use LDFLAGS="-Wl,--no-undefined -Wl,--no-allow-shlib-undefined". This did not solve the problem.

Here is the ldd output for the default case (--as-needed):

        linux-vdso.so.1 (0x00007ffc31b4e000)
        libSyncClient.so.3.26.0-dev => /home/wilsonj/OpenDDS/lib/libSyncClient.so.3.26.0-dev (0x00007fa7a2ae1000)
        libOpenDDS_Dcps.so.3.26.0-dev => /home/wilsonj/OpenDDS/lib/libOpenDDS_Dcps.so.3.26.0-dev (0x00007fa7a1831000)
        libTAO_Valuetype.so.2.5.20 => /home/wilsonj/OpenDDS/ACE_wrappers/lib/libTAO_Valuetype.so.2.5.20 (0x00007fa7a1806000)
        libTAO_PortableServer.so.2.5.20 => /home/wilsonj/OpenDDS/ACE_wrappers/lib/libTAO_PortableServer.so.2.5.20 (0x00007fa7a16c8000)
        libTAO_AnyTypeCode.so.2.5.20 => /home/wilsonj/OpenDDS/ACE_wrappers/lib/libTAO_AnyTypeCode.so.2.5.20 (0x00007fa7a15ba000)
        libTAO.so.2.5.20 => /home/wilsonj/OpenDDS/ACE_wrappers/lib/libTAO.so.2.5.20 (0x00007fa7a13d2000)
        libACE.so.6.5.20 => /home/wilsonj/OpenDDS/ACE_wrappers/lib/libACE.so.6.5.20 (0x00007fa7a11e5000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fa7a1012000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fa7a0ff8000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa7a0fd6000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa7a0e02000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fa7a0cbc000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fa7a0cb6000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fa7a0cac000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fa7a2c17000)

And here is the ldd output for --no-as-needed:

        linux-vdso.so.1 (0x00007ffe465fc000)
        libSyncClient.so.3.26.0-dev => /home/wilsonj/OpenDDS/lib/libSyncClient.so.3.26.0-dev (0x00007f80381b7000)
        libOpenDDS_InfoRepoDiscovery.so.3.26.0-dev => /home/wilsonj/OpenDDS/lib/libOpenDDS_InfoRepoDiscovery.so.3.26.0-dev (0x00007f803813e000)
        libOpenDDS_Tcp.so.3.26.0-dev => /home/wilsonj/OpenDDS/lib/libOpenDDS_Tcp.so.3.26.0-dev (0x00007f80380c5000)
        libOpenDDS_Dcps.so.3.26.0-dev => /home/wilsonj/OpenDDS/lib/libOpenDDS_Dcps.so.3.26.0-dev (0x00007f8036e15000)
        libTAO_BiDirGIOP.so.2.5.20 => /home/wilsonj/OpenDDS/ACE_wrappers/lib/libTAO_BiDirGIOP.so.2.5.20 (0x00007f8036dff000)
        libTAO_PI.so.2.5.20 => /home/wilsonj/OpenDDS/ACE_wrappers/lib/libTAO_PI.so.2.5.20 (0x00007f8036da1000)
        libTAO_CodecFactory.so.2.5.20 => /home/wilsonj/OpenDDS/ACE_wrappers/lib/libTAO_CodecFactory.so.2.5.20 (0x00007f8036d81000)
        libTAO_Valuetype.so.2.5.20 => /home/wilsonj/OpenDDS/ACE_wrappers/lib/libTAO_Valuetype.so.2.5.20 (0x00007f8036d56000)
        libTAO_PortableServer.so.2.5.20 => /home/wilsonj/OpenDDS/ACE_wrappers/lib/libTAO_PortableServer.so.2.5.20 (0x00007f8036c18000)
        libTAO_AnyTypeCode.so.2.5.20 => /home/wilsonj/OpenDDS/ACE_wrappers/lib/libTAO_AnyTypeCode.so.2.5.20 (0x00007f8036b0a000)
        libTAO.so.2.5.20 => /home/wilsonj/OpenDDS/ACE_wrappers/lib/libTAO.so.2.5.20 (0x00007f8036924000)
        libACE.so.6.5.20 => /home/wilsonj/OpenDDS/ACE_wrappers/lib/libACE.so.6.5.20 (0x00007f8036735000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f8036729000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f803671f000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f8036552000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f803640e000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f80363f4000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f80363d2000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f80361fc000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f80382ed000)

The direct dependency of the application on SyncClient explains why adding BiDirGIOP.h solved the problem; the linker did not elide the static initializers, so TAO_BiDirGIOP was loaded and initialized correctly.

The ldd output shows how --as-needed strips dependencies that are loaded under application control. These dependencies can and are loaded but it seems that the static intializers they contain are getting elided.

Conclusion and Call to Action

It seems like --as-needed breaks the service configuration framework in ACE when a manually loaded library depends on other libraries with static initializers. The ld version in question is 2.35.2. This was discovered on Ubuntu and appears to be common to Debian derivatives.

• Has anyone else seen this issue, and if so, what was the workaround?
• Does the service configuration framework need to be tweaked to overcome --as-needed or is it more of a build configuration issue?

[jwillemsen]

ACE service configurator depends on the fact that the static initializers do run. The 'workaround' to add an explicit include is something that is done in a lot of ACE/TAO tests for the static builds, but that shouldn't be necessary in a dynamic build.

The best seems to not use --as-needed. Ubuntu/Debian isn't tested at all, so when only there they use this flag as default we wouldn't have seen it. Maybe the best is to add a small ACE test to detect this problem automatically and on the platform(s) where this is a problem add -Wl,--no-as-needed to the linker flags.

[jrw972]

Reasonable. I'm working on a minimal example to reproduce the problem. I'll share that once I have it working as we can go from there.

[jrw972]

Update

Unsurprisingly, attempting to reproduce the problem yielded a better understanding of it.

When using -Wl,--no-as-needed, the sequence of events is as follows:

1. Executable starts.
2. TAO_BiDirGIOP static initializer executes (see TAO_Requires_BiDirGIOP_Initializer) which loads the module and registers BiDirGIOP. TAO_BiDirGIOP_Loader::init is not called.
3. ORB_init is called.
4. ORB_init ends up calling TAO::ORB::open_global_services which calls TAO_BiDirGIOP_Loader::init which loads the policy.
5. ORB_init is called again, TAO::ORB::open_global_services returns early because it has already been called.
6. Uses of BiDirGIOP succeed because policy is loaded.

When using --as-needed, the sequence changes to:

1. Executable starts.
2. ORB_init is called.
3. TAO::ORB::open_global_services is called but doesn't call TAO_BiDirGIOP_Loader::init because BiDirGIOP is not loaded or registered.
4. Dynamic module loading causes TAO_BiDirGIOP static initializer to execute (see TAO_Requires_BiDirGIOP_Initializer) which loads the module and registers BiDirGIOP. TAO_BiDirGIOP_Loader::init is not called.
5. ORB_init is called. TAO::ORB::open_global_services returns early because it has already been called.
6. Attempts to use BiDirGIOP fail because the policy is not loaded.

The main point I was missing was that the call to init is not called from the static initializer.
For the code in question, --as-needed causes the execution of the static initializer that sets up the eventual call to init to be executed after TAO::ORB::open_global_services.
To fix it, I added the snippet that calls init for BiDirGIOP to OpenDDS_InfoRepoDiscovery.

Summary

• TAO_BiDirGIOP_Loader::init must be called before using BiDirGIOP policies.
• ACE/TAO uses static initializers to populate a registry of loadable modules. This registry can then be invoked to load the dynamic module. The init method is not called.
• ACE/TAO is built on the assumption that all static initializers are executed before and in anticipation of global initialization like TAO::ORB::open_global_services which calls TAO_BiDirGIOP_Loader::init.
• Before --as-needed, the linker always included the static initializers for specified linking dependencies. This created the possibility of a plugin architecture where the executable does not directly depend on the loadable module. The necessary infrastructure to load the module was carried by the static initializer. The plugin could be loaded on-demand.
• After --as-needed, the linker can prune dependencies when the code in the dependency is not used. Thus, a plugin and its dependencies may get pruned away. The plugin may still be loaded with the consequence that its static initializers are executed after the dynamic module has been loaded instead of before. If this occurs after TAO::ORB::open_global_services, then initialization methods like TAO_BiDirGIOP_Loader::init will not be called.

Possible Next Steps

As stated earlier, once I understood more about ACE/TAO, adding a little initialization code to solve the problem was straightforward.

One possibility is using -Wl,--no-as-needed for all modules that depend on global initialization like TAO::ORB::open_global_services. This would force the static initializers to be present in the executable so they would be registered for TAO::ORB::open_global_services.

Another possibility is to update the global initialization code to respect things that have been loaded since the previous execution.

In either case, this probably not going to affect many users. Perhaps the best thing is just documentation.

[mitza-oci]

Was the use of --as-needed changed at some point - and if so do you have the PR#?

[jrw972]

The Debian/Ubuntu distributions changed the default behavior of ld so the change was external.