Building Horovod with latest tensorflow 2.15.0

[supercharleszhu]

Hi Horovod team, we are trying to build horovod with latest tensorflow 2.15.0, seems that some symbol is missing when importing

Steps to reproduce:
in the build environment:

pip3 install tensorflow==${tensorflow_VERSION}
python3.10 -c "import tensorflow; print(tensorflow.__version__)"

cd horovod
HOROVOD_WITH_GLOO=1 HOROVOD_WITH_TENSORFLOW=1 HOROVOD_WITHOUT_PYTORCH=1 HOROVOD_WITHOUT_MXNET=1 HOROVOD_WITHOUT_MPI=1 HOROVOD_CPU_OPERATIONS=GLOO HOROVOD_GPU_OPERATIONS=NCCL python3 setup.py bdist_wheel

pip3 install dist/*.whl
python -c "import horovod.tensorflow as hvd"

Error:

  File "/home/chzhu/chzhu_tb_investigation/env_310/lib/python3.10/site-packages/horovod/tensorflow/mpi_ops.py", line 50, in <module>
    MPI_LIB = _load_library('mpi_lib' + get_ext_suffix())
  File "/home/chzhu/chzhu_tb_investigation/env_310/lib/python3.10/site-packages/horovod/tensorflow/mpi_ops.py", line 45, in _load_library
    library = load_library.load_op_library(filename)
  File "/home/chzhu/chzhu_tb_investigation/env_310/lib/python3.10/site-packages/tensorflow/python/framework/load_library.py", line 54, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /home/chzhu/chzhu_tb_investigation/env_310/lib/python3.10/site-packages/horovod/tensorflow/mpi_lib.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZNK10tensorflow8OpKernel11TraceStringERKNS_15OpKernelContextEb

Do we have current plan for updating to latest tensorflow? Let me know if I can contritbute to this effort. Thanks!

[chongxiaoc]

@EnricoMi @maxhgerlach Do you have experience of trying TF 2.15?
Is it related with TF head CI failure as well?

[supercharleszhu]

Turns out that the issue is coming from the different GCC we are used when compiling tensorflow VS horovod. Closing the issue for now.