Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

undefined symbol in libaccp-clang.so #1459

Closed
FluxusMagna opened this issue May 14, 2024 · 8 comments
Closed

undefined symbol in libaccp-clang.so #1459

FluxusMagna opened this issue May 14, 2024 · 8 comments
Labels
bug Something isn't working

Comments

@FluxusMagna
Copy link

FluxusMagna commented May 14, 2024

Bug summary
I'm trying to build gromacs with support for my AMD gpu with ACCP, but when CMAKE checks for a valid AdaptiveCpp/hipSYCL compiler I get the error:

 error: unable to load plugin '/usr/bin/../lib/libacpp-clang.so':
 '/usr/bin/../lib/libacpp-clang.so: undefined symbol:
 _ZTIN4llvm2cl18GenericOptionValueE'
  

My setup and how to reproduce

I'm running an up to date arch-linux system with rocm 6.0.2-2 from the arch repo and the adaptivecpp-rocm-git 24.02.0+17.r2664.20240325.37d1dcd7-1 aur package installed.

To build gromacs I run a script with the version (ie, 2024.2) as the argument

wget https://ftp.gromacs.org/gromacs/gromacs-$1.tar.gz
rm -rf gromacs-$1
tar xfz gromacs-$1.tar.gz
cd gromacs-$1
mkdir build
cd build

ROCM_PATH="/opt/rocm"
cmake .. -DGMXAPI=off \
         -DCMAKE_C_COMPILER=${ROCM_PATH}/llvm/bin/clang \
         -DCMAKE_CXX_COMPILER=${ROCM_PATH}/llvm/bin/clang++ \
         -DLLVM_DIR=${ROCM_PATH}/llvm/lib/cmake/llvm/ \
         -DGMX_GPU=SYCL -DGMX_SYCL=ACPP -DHIPSYCL_TARGETS='hip:gfx1030'

Expected behavior
The requested symbol should not be undefined

Optional additional diagnostic information

syclcc output:

acpp warning: syclcc is deprecated; please use acpp instead.
acpp [AdaptiveCpp compilation driver], Copyright (C) 2018-2023 Aksel Alpay and the AdaptiveCpp project
  AdaptiveCpp version: 24.02.0+git.d57b9f58.20240514.branch.develop
  Installation root: /usr
  Plugin LLVM version: 17, can accelerate CPU: True
  Available runtime backends:
     librt-backend-omp.so
     librt-backend-hip.so
     librt-backend-ocl.so


Full configuration [can be overridden using environment variables or command line arguments]:
    version-major: 24
    version-minor: 02
    version-patch: 0
    version-suffix: +git.d57b9f58.20240514.branch.develop
    plugin-llvm-version-major: 17
    plugin-with-cpu-acceleration: true
    default-clang: /usr/bin/clang++
    default-targets: generic
    default-cpu-cxx: /usr/bin/c++
    default-rocm-path: /opt/rocm
    default-use-bootstrap-mode: false
    default-is-dryrun: false
    default-use-accelerated-cpu: true
    default-clang-include-path: /usr/lib/clang/17/include/..
    default-sequential-link-line: -L/usr/lib -lboost_context -lboost_fiber -Wl,-rpath=/usr/lib
    default-sequential-cxx-flags: -I/usr/include -D_ENABLE_EXTENDED_ALIGNED_STORAGE
    default-omp-link-line: -L/usr/lib -lboost_context -lboost_fiber -Wl,-rpath=/usr/lib -fopenmp
    default-omp-cxx-flags: -I/usr/include -fopenmp -D_ENABLE_EXTENDED_ALIGNED_STORAGE
    default-is-explicit-multipass: false
    default-save-temps: false
    default-rocm-link-line: -Wl,-rpath=$HIPSYCL_ROCM_PATH/lib -Wl,-rpath=$HIPSYCL_ROCM_PATH/hip/lib -L/opt/rocm/lib -L/opt/rocm/hip/lib -lamdhip64
    default-rocm-cxx-flags: -isystem $HIPSYCL_PATH/include/AdaptiveCpp/hipSYCL/std/hiplike -isystem /usr/lib/clang/17/include/.. -U__FLOAT128__ -U__SIZEOF_FLOAT128__ -I$HIPSYCL_ROCM_PATH/include -I$HIPSYCL_ROCM_PATH/include --rocm-device-lib-path=$HIPSYCL_ROCM_PATH/amdgcn/bitcode --rocm-path=$HIPSYCL_ROCM_PATH -fhip-new-launch-api -mllvm -amdgpu-early-inline-all=true -mllvm -amdgpu-function-calls=false -D__HIP_ROCclr__
@FluxusMagna FluxusMagna added the bug Something isn't working label May 14, 2024
@illuhad
Copy link
Collaborator

illuhad commented May 14, 2024

Hi!

The AUR packages are maintained by third parties, and not officially supported. I don't know what state they are currently in. Last time I checked, I would rate the packages as "good first effort, not for production" but maybe they have improved.

The ABI issue that you see generally indicates that you are trying to load the AdaptiveCpp clang plugin into a different clang/LLVM than it was compiled for. This means that something in your AdaptiveCpp or Gromacs build process is broken.

Can you please try compiling a simple test program? E.g. this one: https://github.com/AdaptiveCpp/AdaptiveCpp/blob/develop/doc/examples.md
Gromacs currently still uses the old SMCP compilation flow, so you will need to use acpp --acpp-targets=hip:gfxXYZ test.cpp. You need to replace gfxXYZ with your GPU architecture, which acpp-info will tell you.

Otherwise, perhaps @al42and from the Gromacs team might have some insights as to how this error might occur in the Gromacs build process specifically.

@FluxusMagna
Copy link
Author

FluxusMagna commented May 15, 2024

Thanks for the information!
I tried compiling the test program and got this, which doesn't tell me much.

$ acpp -v -o test test.cpp --acpp-targets=hip:gfx1030
acpp warning: No optimization flag was given, optimizations are disabled by default. Performance may be degraded. Compile with e.g. -O2/-O3 to enable optimizations.
clang version 17.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-pc-linux-gnu/11.3.0
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-pc-linux-gnu/12.3.0
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-pc-linux-gnu/13.2.1
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-pc-linux-gnu/14.1.1
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-pc-linux-gnu/7.5.0
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-pc-linux-gnu/8.4.0
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-pc-linux-gnu/9.3.0
Found candidate GCC installation: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/11.3.0
Found candidate GCC installation: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/12.3.0
Found candidate GCC installation: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1
Found candidate GCC installation: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.1.1
Found candidate GCC installation: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/7.5.0
Found candidate GCC installation: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/8.4.0
Found candidate GCC installation: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/9.3.0
Selected GCC installation: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.1.1
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
Found HIP installation: /opt/rocm, version 6.0.32831
 "/usr/bin/clang++" -cc1 -triple amdgcn-amd-amdhsa -aux-triple x86_64-pc-linux-gnu -emit-obj -mrelax-all -dumpdir test- -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name test.cpp -mrelocation-model pic -pic-level 2 -fhalf-no-semantic-interposition -mframe-pointer=all -fno-rounding-math -mconstructor-aliases -aux-target-cpu x86-64 -fcuda-is-device -mllvm -amdgpu-internalize-symbols -fcuda-allow-variadic-functions -fvisibility=hidden -fapply-global-visibility-to-externs -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/hip.bc -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/ocml.bc -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/ockl.bc -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/oclc_daz_opt_off.bc -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/oclc_unsafe_math_off.bc -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/oclc_finite_only_off.bc -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/oclc_correctly_rounded_sqrt_on.bc -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/oclc_wavefrontsize64_off.bc -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/oclc_isa_version_1030.bc -mlink-builtin-bitcode /opt/rocm/amdgcn/bitcode/oclc_abi_version_400.bc -target-cpu gfx1030 -debugger-tuning=gdb -v -resource-dir /usr/lib/clang/17 -internal-isystem /usr/lib/clang/17/include/cuda_wrappers -idirafter /opt/rocm/include -include __clang_hip_runtime_wrapper.h -isystem /usr/bin/../include/AdaptiveCpp -isystem /usr/bin/../include/AdaptiveCpp/hipSYCL/std/hiplike -isystem /usr/lib/clang/17/include/.. -D __OPENSYCL__ -D __HIPSYCL__ -D __ADAPTIVECPP__ -D __ACPP__ -U __FLOAT128__ -U __SIZEOF_FLOAT128__ -I /opt/rocm/include -I /opt/rocm/include -D __HIP_ROCclr__ -D __HIPSYCL_ENABLE_HIP_TARGET__ -D __HIPSYCL_CLANG__ -D __HIPSYCL_ENABLE_OMPHOST_TARGET__ -I /usr/include -D _ENABLE_EXTENDED_ALIGNED_STORAGE -internal-isystem /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../include/c++/14.1.1 -internal-isystem /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../include/c++/14.1.1/x86_64-pc-linux-gnu -internal-isystem /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../include/c++/14.1.1/backward -internal-isystem /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../include/c++/14.1.1 -internal-isystem /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../include/c++/14.1.1/x86_64-pc-linux-gnu -internal-isystem /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../include/c++/14.1.1/backward -internal-isystem /usr/lib/clang/17/include -internal-isystem /usr/local/include -internal-isystem /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../x86_64-pc-linux-gnu/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /usr/lib/clang/17/include -internal-isystem /usr/local/include -internal-isystem /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../x86_64-pc-linux-gnu/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -std=c++17 -fdeprecated-macro -fno-autolink -fdebug-compilation-dir=/home/gusten/AdaptiveCpp/doc -ferror-limit 19 -fhip-new-launch-api -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fcolor-diagnostics -load /usr/bin/../lib/libacpp-clang.so -fpass-plugin=/usr/bin/../lib/libacpp-clang.so -mllvm -amdgpu-early-inline-all=true -mllvm -amdgpu-function-calls=false -cuid=42d9534ca6564ffe -fcuda-allow-variadic-functions -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /tmp/test-gfx1030-384b5e.o -x hip test.cpp
clang -cc1 version 17.0.6 based upon LLVM 17.0.6 default target x86_64-pc-linux-gnu
ignoring nonexistent directory "/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../x86_64-pc-linux-gnu/include"
ignoring nonexistent directory "/include"
ignoring nonexistent directory "/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../x86_64-pc-linux-gnu/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/opt/rocm/include"
ignoring duplicate directory "/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../include/c++/14.1.1"
ignoring duplicate directory "/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../include/c++/14.1.1/x86_64-pc-linux-gnu"
ignoring duplicate directory "/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../include/c++/14.1.1/backward"
ignoring duplicate directory "/usr/include"
  as it is a non-system directory that duplicates a system directory
ignoring duplicate directory "/usr/lib/clang/17/include"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/usr/include"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/usr/lib/clang/17/include"
ignoring duplicate directory "/usr/include"
ignoring duplicate directory "/opt/rocm/include"
  as it is a non-system directory that duplicates a system directory
#include "..." search starts here:
#include <...> search starts here:
 /usr/bin/../include/AdaptiveCpp
 /usr/bin/../include/AdaptiveCpp/hipSYCL/std/hiplike
 /usr/lib/clang/17/include/..
 /usr/lib/clang/17/include/cuda_wrappers
 /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../include/c++/14.1.1
 /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../include/c++/14.1.1/x86_64-pc-linux-gnu
 /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.1.1/../../../../include/c++/14.1.1/backward
 /usr/lib/clang/17/include
 /usr/local/include
 /usr/include
 /opt/rocm/include
End of search list.
 "/usr/bin/clang-offload-bundler" -unbundle -type=a -input=/usr/bin/../lib//libboost_context.a -targets=hip-amdgcn-amd-amdhsa-gfx1030 -output=/tmp/libbc-boost_context-amdgcn-gfx1030-52c2a6.a -allow-missing-bundles -hip-openmp-compatible
 "/usr/bin/clang-offload-bundler" -unbundle -type=a -input=/usr/bin/../lib//libboost_fiber.a -targets=hip-amdgcn-amd-amdhsa-gfx1030 -output=/tmp/libbc-boost_fiber-amdgcn-gfx1030-44d9dc.a -allow-missing-bundles -hip-openmp-compatible
 "lld" -flavor gnu -m elf64_amdgpu --no-undefined -shared -plugin-opt=-amdgpu-internalize-symbols -plugin-opt=mcpu=gfx1030 --whole-archive -o /tmp/test-gfx1030-88da8c.out /tmp/test-gfx1030-384b5e.o /tmp/libbc-boost_context-amdgcn-gfx1030-52c2a6.a /tmp/libbc-boost_fiber-amdgcn-gfx1030-44d9dc.a --no-whole-archive
clang++: error: unable to execute command: Executable "lld" doesn't exist!
clang++: error: amdgcn-link command failed with exit code 1 (use -v to see invocation)

In the gromacs example though, I'm using the clang provided by rocm which is the recommended way according to the gromacs documentation. The rocm package is apparently flagged out of date, but this could just mean there is a newer version available, not that it's broken.

If the clang version it was compiled for is the problem, how do I change it? Is it some cmake variable?

@illuhad
Copy link
Collaborator

illuhad commented May 17, 2024

I tried compiling the test program and got this, which doesn't tell me much.

It does tell you that AdaptiveCpp was built against /usr/bin/clang++, which however is unable to build HIP binaries because you are missing lld (missing package probably?). Try installing lld.

In the gromacs example though, I'm using the clang provided by rocm which is the recommended way according to the gromacs documentation.

It is true that Gromacs recommends this (and I am sure they have good reasons), however, our AdaptiveCpp documentation actually discourages this configuration and only partially supports this (also for good reasons). I'm not sure whether the reasoning for recommending building against ROCm clang applies to end users running things on a desktop system with Arch Linux as well. On an HPC cluster, software stacks and requirements can be quite different compared to an end user system.

From the AdaptiveCpp point of view, you might have an easier time just building against your system clang/LLVM, which the package seems to be doing as well.
However, I don't know whether the Gromacs build system somehow enforces ROCm clang to be used, which could explain why you get this error.

You can tell AdaptiveCpp which LLVM to build against using -DLLVM_DIR and -DCLANG_EXECUTABLE_PATH as described here: https://github.com/AdaptiveCpp/AdaptiveCpp/blob/develop/doc/install-llvm.md#pointing-adaptivecpp-to-the-right-llvm
This requires however configuring it by yourself and not using the package.

There is also a way to override the clang that it tries to use using the ACPP_CLANG environment variable, however you should in practice almost never do this - using any clang other than the one that you have built AdaptiveCpp for is going to result in errors like you are seeing.

@FluxusMagna
Copy link
Author

FluxusMagna commented May 17, 2024

It seems you are correct. I didn't consider that lld could actually be missing since I've built a lot of software on this machine, and I'm not sure how it would have been uninstalled. Perhaps it was included in some other package previously, and is now separate.

I tried building Gromacs with the system clang and it indeed compiles, and passes the tests. I guess I'll see how it works later, but I consider this issue closed for now anyway.

Thanks a lot!

@al42and
Copy link
Contributor

al42and commented May 20, 2024

It is true that Gromacs recommends this (and I am sure they have good reasons), however, our AdaptiveCpp documentation actually discourages this configuration and only partially supports this (also for good reasons). I'm not sure whether the reasoning for recommending building against ROCm clang applies to end users running things on a desktop system with Arch Linux as well.

The rationale for recommending amdclang from GROMACS side is two-fold:

  1. Testing. With amdclang we have "(GROMACS version) × (AdaptiveCpp version) × (ROCm version)" combinatorics. And there are already many subtle ways things interact (e.g. while Lazily construct multi-queue executors on all backends #1318 reduced the resource usage on AMD, turns out that in some cases having idle hipStream's on device can improve performance due to the way hipStream's are mapped to native hardware streams and the way these hardware streams behave; or simply performance regressions with newer compilers). This is already hard to test for correctness/performance; adding "× (LLVM version)" into the mix would make things even worse.

  2. Availability of correct LLVM. There are many ways to build LLVM/Clang, and I've seen installations without libLLVM.so or without the ability to emit AMDGCN/NVPTX. amdclang is not great (does not support SSCP, for example), but it is predictable.

Other than that, there is nothing wrong with using GROMACS with mainline Clang. It is tested to work. Apparently, the AdaptiveCpp package in AUR is built against system Clang; in this case, it is reasonable to also use it for GROMACS. As I gather, this is what worked for you in the end. So, all good!

@illuhad
Copy link
Collaborator

illuhad commented May 22, 2024

Thanks for sharing your insights! One small remark:

amdclang is not great (does not support SSCP, for example), but it is predictable.

I think this is only the case if you limit the compiler interaction with amdclang to a minimum (which generally is the case for CUDA/HIP compilation flows). As soon as you want to do something more involved with it (e.g. SSCP), it is far less predictable and generally unstable.

It's true that LLVM is complex and there are lots of ways an installation can be configured. But we do how exactly it has to be built, including example cmake invocation.

@al42and
Copy link
Contributor

al42and commented May 22, 2024

I think this is only the case if you limit the compiler interaction with amdclang to a minimum (which generally is the case for CUDA/HIP compilation flows).

True. We're just recommending it for narrow GROMACS use case on AMD GPUs; in the NVIDIA section of the manual, we recommend mainline LLVM.

I guess we need to explicitly mention that if a pre-built AdaptiveCpp is installed, then a matching Clang should be used.

As soon as you want to do something more involved with it (e.g. SSCP), it is far less predictable

It's quite predictable in not supporting SSCP at all, last time I checked :)

But we do how exactly it has to be built, including example cmake invocation.

I've been at Cray User Group meeting a couple weeks ago, and a few site admins complained to me that installing AdaptiveCpp is already a hassle they'd very much prefer to avoid. Pretty sure that a suggestion to first build LLVM won't be appreciated (not only making sure that the suggested CMake command actually works with their environment, but also writing module files, and later being responsible for it, upgrading it when the software stack is updated, etc). For end-users: if you think a typical biophysics undergrad knows how to find the "subdirectory containing the LLVM cmake files" (or what most of those words mean), you're waaay too optimistic :)

@illuhad
Copy link
Collaborator

illuhad commented May 22, 2024

I've been at Cray User Group meeting a couple weeks ago, and a few site admins complained to me that installing AdaptiveCpp is already a hassle they'd very much prefer to avoid. Pretty sure that a suggestion to first build LLVM won't be appreciated (not only making sure that the suggested CMake command actually works with their environment, but also writing module files, and later being responsible for it, upgrading it when the software stack is updated, etc). For end-users: if you think a typical biophysics undergrad knows how to find the "subdirectory containing the LLVM cmake files" (or what most of those words mean), you're waaay too optimistic :)

The example cmake invocation on https://github.com/AdaptiveCpp/AdaptiveCpp/blob/develop/doc/install-llvm.md was actually used multiple times on different Cray systems. I'm not aware of any problems that might occur there. However, I think it's true that there are some things we can do to help, especially around packaging (e.g. spack).
That said, I am also surprised to hear this complaint since I don't believe that AdaptiveCpp is substantially more difficult to install compared to other similar compiler projects when building from source - LLVM with OpenMP offload support, open source DPC++ etc.

Regarding the biophysics undergrad, I think there is a discrepancy between the target audience of AdaptiveCpp, which is developers of accelerated software, and end users of that software which then might be your undergrad student. Bridging this gap probably is where traditionally system administrators with the necessary expertise or packagers come into play. But I'm not sure this is an AdaptiveCpp-specific problem :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants