Build catboost python package with custom glibc

[egorvasilevich]

Problem: Hello there! Do you tried to build catboost python package with custom glibc? I want to use python catboost lib in secure way via Occlum, but Occlum have its own custom toolchain and glibc and i don't know how to correctly build catboost with theese toolchain. Can you help me? May be you know where i need to pass path to glibc directory or where i need to set up flags for clang
So much thanks!
Versions:

• setuptools: 69.2.0
• jupyterlab: 3.6.7
• conan: 1.64.0
• wheel: 0.34.2-1
• cmake: 3.29.2
• ninja: 1.10.0
• clang: 14.0.6

catboost version: 1.2.4
Operating System: Ubuntu 20.04.6
CPU: i5 1155G7
GPU: -

[andrey-khropov]

1. First, you have to build libraries provided by Conan that CatBoost depends on with a custom conan host profile so they will link against custom glibc as well. See the relevant conan documentation.

2. After that, if you want to modify the way binary extension so library for python package is built you can do it using either:

   1. By modifying arguments to build_native.build call in setup.py here

   or

   2. Build binary artifact target _catboost first using build_native.py or by calling cmake, ninja directly and calling setup.py with --prebuilt-extensions-build-root-dir option.

   In either case you'd probably want to use custom cmake toolchain file. The default one that is used on Linux is clang.toolchain. You can either modify it directly or clone it to a new file and make modifications there. Custom toolchain can be passed to build_native using cmake-tagret-toolchain option or to cmake directly using CMAKE_TOOLCHAIN_FILE variable

   You might want to modify CMake variables there such as CMAKE_SHARED_LINKER_FLAGS and CMAKE_C_STANDARD_LIBRARIES, CMAKE_CXX_STANDARD_LIBRARIES.

   I don't know specifics about how Occlum works, so I can't say exactly what flags are appropriate.

[andrey-khropov]

BTW

catboost version: 1.2.4

Such version does not exist, it has been skipped due to issues during the release process.

[egorvasilevich]

Thank you for the reply. I made the attempt to patch clang.toolchain with CMAKE_EXE_LINKER_FLAGS and CMAKE_SHARED_LINKER_FLAGS like this

set(CMAKE_C_COMPILER clang)
set(CMAKE_CXX_COMPILER clang++)
 
set(CMAKE_C_STANDARD_LIBRARIES "-lc -lm")
set(CMAKE_CXX_STANDARD_LIBRARIES "-lc -lm")
 
set(CMAKE_CUDA_COMPILER nvcc)
set(CMAKE_CUDA_HOST_COMPILER clang-12)
 
set(CMAKE_CUDA_FLAGS_INIT "--allow-unsupported-compiler")
 
if (NOT APPLE)
  set(CMAKE_EXE_LINKER_FLAGS -fuse-ld=lld -Wl,--rpath=/opt/occlum/glibc/lib -Wl,--dynamic-linker=/opt/occlum/glibc/lib/ld-linux-x86-64.so.2)
  set(CMAKE_SHARED_LINKER_FLAGS -fuse-ld=lld -Wl,--rpath=/opt/occlum/glibc/lib -Wl,--dynamic-linker=/opt/occlum/glibc/lib/ld-linux-x86-64.so.2)
endif()
 
set(ENV{CC} clang)
set(ENV{CXX} clang++)
 
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)

Next step — build catboost with
command python3.8 setup.py bdist_wheel --no-widget
Catboost python whl was builded successfully, but when I tried to run it in occlum, I face the same error

Catch task error in thread: Failed execute script\n --> at crates/task_executors/python/src/lib.rs:116\nCaused by:\n  CatBoostError: (Function not implemented) util/system/file.cpp:857: can't open \"/proc/self/statm\" with mode RdOnly|Seq (0x00000028)\n

As I understand conan uses the same compiler and linker settings as a cmake. According to your reply patching clang.toolchain is enough to build catboost with custom glibc, but the result is unsuccessful.

Other attempts to find the workaround:

1. Tried to swap glibc in occlum dev container with occlum custom glibc, but it caused a container OS broken.
2. Tried to swap gcc libs (/usr/lib/gcc/x86_64-linux-gnu/) with occlum gcc libs, but catboost build crashes on the early steps (on compiler test, error - ld.lld: error: unable to find library -lgcc_s)

Additionally,
Occlum toolchain does not have custom clang, but has custom musl-gcc (with occlum-gcc, occlum-g++ and occlum-ld). Also musl-gcc includes custom glibc where main changes are redirecting syscalls, supporting posix_spawn syscall, modified vdso to calling syscalls.

We tried to link that glibc to clang in clang.toolchain file, but that does not work (as described above). As a result, catboost used standard glibc.

As I see, we need to file change request to make Catboost to be supported by Occlum LibOS environment. Options:

1. using musl-gcc for building catboost native artifact
2. supporting TEE platforms (aka Intel SGX) and you could release version with supporting Occlum LibOS (looks like the only option for now)

Would be happy if you highlighted any other workarounds.