Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

No module named 'packaging' when installing Horovod #4003

Open
flixxox opened this issue Nov 10, 2023 · 9 comments · May be fixed by #4016
Open

No module named 'packaging' when installing Horovod #4003

flixxox opened this issue Nov 10, 2023 · 9 comments · May be fixed by #4016
Labels
bug

Comments

@flixxox
Copy link

flixxox commented Nov 10, 2023

Hi,
I am getting the error ModuleNotFoundError: No module named 'packaging' when trying to install horovod.

I am running the following commands:

python3 -m venv ~/lib/test-venv
source ~/lib/test-venv/bin/activate
pip3 install --upgrade pip
pip3 install numpy
pip3 install https://download.pytorch.org/whl/cu118/torch-2.0.1%2Bcu118-cp311-cp311-linux_x86_64.whl
pip3 install packaging
HOROVOD_WITH_PYTORCH=1 pip3 install --no-cache-dir horovod[pytorch]

My setup is:

  • Ubuntu 20.04.2 LTS
  • PyTorch 2.0.1
  • Horovod 0.28.1
  • MPI 5.0.0
  • NCCL 2.19.3
  • Python 3.11.4
  • GCC 7.5.0
  • CMake 3.28.0

I think my issue relates to these issues but they couldn't help me (#3744, #3697).

Full error message:

(test-venv) fschmidt@ccx-00:~/lib$ HOROVOD_WITH_PYTORCH=1 pip3 install --no-cache-dir horovod[pytorch]
Collecting horovod[pytorch]
  Downloading horovod-0.28.1.tar.gz (3.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.5/3.5 MB 130.2 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting cloudpickle (from horovod[pytorch])
  Downloading cloudpickle-3.0.0-py3-none-any.whl.metadata (7.0 kB)
Collecting psutil (from horovod[pytorch])
  Downloading psutil-5.9.6-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (21 kB)
Collecting pyyaml (from horovod[pytorch])
  Downloading PyYAML-6.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB)
Requirement already satisfied: packaging in ./test-venv/lib/python3.11/site-packages (from horovod[pytorch]) (23.2)
Collecting cffi>=1.4.0 (from horovod[pytorch])
  Downloading cffi-1.16.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.5 kB)
Requirement already satisfied: torch in ./test-venv/lib/python3.11/site-packages (from horovod[pytorch]) (2.0.1+cu118)
Collecting pycparser (from cffi>=1.4.0->horovod[pytorch])
  Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.7/118.7 kB 76.6 MB/s eta 0:00:00
Requirement already satisfied: filelock in ./test-venv/lib/python3.11/site-packages (from torch->horovod[pytorch]) (3.13.1)
Requirement already satisfied: typing-extensions in ./test-venv/lib/python3.11/site-packages (from torch->horovod[pytorch]) (4.8.0)
Requirement already satisfied: sympy in ./test-venv/lib/python3.11/site-packages (from torch->horovod[pytorch]) (1.12)
Requirement already satisfied: networkx in ./test-venv/lib/python3.11/site-packages (from torch->horovod[pytorch]) (3.2.1)
Requirement already satisfied: jinja2 in ./test-venv/lib/python3.11/site-packages (from torch->horovod[pytorch]) (3.1.2)
Requirement already satisfied: triton==2.0.0 in ./test-venv/lib/python3.11/site-packages (from torch->horovod[pytorch]) (2.0.0)
Requirement already satisfied: cmake in ./test-venv/lib/python3.11/site-packages (from triton==2.0.0->torch->horovod[pytorch]) (3.27.7)
Requirement already satisfied: lit in ./test-venv/lib/python3.11/site-packages (from triton==2.0.0->torch->horovod[pytorch]) (17.0.4)
Requirement already satisfied: MarkupSafe>=2.0 in ./test-venv/lib/python3.11/site-packages (from jinja2->torch->horovod[pytorch]) (2.1.3)
Requirement already satisfied: mpmath>=0.19 in ./test-venv/lib/python3.11/site-packages (from sympy->torch->horovod[pytorch]) (1.3.0)
Downloading cffi-1.16.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (464 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 464.8/464.8 kB 168.1 MB/s eta 0:00:00
Downloading cloudpickle-3.0.0-py3-none-any.whl (20 kB)
Downloading psutil-5.9.6-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (283 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 283.6/283.6 kB 261.8 MB/s eta 0:00:00
Downloading PyYAML-6.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (757 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 757.7/757.7 kB 262.0 MB/s eta 0:00:00
Building wheels for collected packages: horovod
  Building wheel for horovod (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for horovod (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [242 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-311
      creating build/lib.linux-x86_64-cpython-311/horovod
      copying horovod/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod
      creating build/lib.linux-x86_64-cpython-311/horovod/common
      copying horovod/common/exceptions.py -> build/lib.linux-x86_64-cpython-311/horovod/common
      copying horovod/common/process_sets.py -> build/lib.linux-x86_64-cpython-311/horovod/common
      copying horovod/common/basics.py -> build/lib.linux-x86_64-cpython-311/horovod/common
      copying horovod/common/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/common
      copying horovod/common/elastic.py -> build/lib.linux-x86_64-cpython-311/horovod/common
      copying horovod/common/util.py -> build/lib.linux-x86_64-cpython-311/horovod/common
      creating build/lib.linux-x86_64-cpython-311/horovod/tensorflow
      copying horovod/tensorflow/mpi_ops.py -> build/lib.linux-x86_64-cpython-311/horovod/tensorflow
      copying horovod/tensorflow/functions.py -> build/lib.linux-x86_64-cpython-311/horovod/tensorflow
      copying horovod/tensorflow/gradient_aggregation_eager.py -> build/lib.linux-x86_64-cpython-311/horovod/tensorflow
      copying horovod/tensorflow/gradient_aggregation.py -> build/lib.linux-x86_64-cpython-311/horovod/tensorflow
      copying horovod/tensorflow/sync_batch_norm.py -> build/lib.linux-x86_64-cpython-311/horovod/tensorflow
      copying horovod/tensorflow/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/tensorflow
      copying horovod/tensorflow/elastic.py -> build/lib.linux-x86_64-cpython-311/horovod/tensorflow
      copying horovod/tensorflow/util.py -> build/lib.linux-x86_64-cpython-311/horovod/tensorflow
      copying horovod/tensorflow/compression.py -> build/lib.linux-x86_64-cpython-311/horovod/tensorflow
      creating build/lib.linux-x86_64-cpython-311/horovod/keras
      copying horovod/keras/callbacks.py -> build/lib.linux-x86_64-cpython-311/horovod/keras
      copying horovod/keras/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/keras
      copying horovod/keras/elastic.py -> build/lib.linux-x86_64-cpython-311/horovod/keras
      creating build/lib.linux-x86_64-cpython-311/horovod/torch
      copying horovod/torch/optimizer.py -> build/lib.linux-x86_64-cpython-311/horovod/torch
      copying horovod/torch/mpi_ops.py -> build/lib.linux-x86_64-cpython-311/horovod/torch
      copying horovod/torch/functions.py -> build/lib.linux-x86_64-cpython-311/horovod/torch
      copying horovod/torch/sync_batch_norm.py -> build/lib.linux-x86_64-cpython-311/horovod/torch
      copying horovod/torch/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/torch
      copying horovod/torch/compression.py -> build/lib.linux-x86_64-cpython-311/horovod/torch
      creating build/lib.linux-x86_64-cpython-311/horovod/runner
      copying horovod/runner/task_fn.py -> build/lib.linux-x86_64-cpython-311/horovod/runner
      copying horovod/runner/mpi_run.py -> build/lib.linux-x86_64-cpython-311/horovod/runner
      copying horovod/runner/run_task.py -> build/lib.linux-x86_64-cpython-311/horovod/runner
      copying horovod/runner/gloo_run.py -> build/lib.linux-x86_64-cpython-311/horovod/runner
      copying horovod/runner/launch.py -> build/lib.linux-x86_64-cpython-311/horovod/runner
      copying horovod/runner/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/runner
      copying horovod/runner/js_run.py -> build/lib.linux-x86_64-cpython-311/horovod/runner
      creating build/lib.linux-x86_64-cpython-311/horovod/data
      copying horovod/data/data_loader_base.py -> build/lib.linux-x86_64-cpython-311/horovod/data
      copying horovod/data/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/data
      creating build/lib.linux-x86_64-cpython-311/horovod/ray
      copying horovod/ray/driver_service.py -> build/lib.linux-x86_64-cpython-311/horovod/ray
      copying horovod/ray/elastic_v2.py -> build/lib.linux-x86_64-cpython-311/horovod/ray
      copying horovod/ray/ray_logger.py -> build/lib.linux-x86_64-cpython-311/horovod/ray
      copying horovod/ray/strategy.py -> build/lib.linux-x86_64-cpython-311/horovod/ray
      copying horovod/ray/utils.py -> build/lib.linux-x86_64-cpython-311/horovod/ray
      copying horovod/ray/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/ray
      copying horovod/ray/elastic.py -> build/lib.linux-x86_64-cpython-311/horovod/ray
      copying horovod/ray/runner.py -> build/lib.linux-x86_64-cpython-311/horovod/ray
      copying horovod/ray/adapter.py -> build/lib.linux-x86_64-cpython-311/horovod/ray
      copying horovod/ray/worker.py -> build/lib.linux-x86_64-cpython-311/horovod/ray
      creating build/lib.linux-x86_64-cpython-311/horovod/_keras
      copying horovod/_keras/callbacks.py -> build/lib.linux-x86_64-cpython-311/horovod/_keras
      copying horovod/_keras/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/_keras
      copying horovod/_keras/elastic.py -> build/lib.linux-x86_64-cpython-311/horovod/_keras
      creating build/lib.linux-x86_64-cpython-311/horovod/spark
      copying horovod/spark/mpi_run.py -> build/lib.linux-x86_64-cpython-311/horovod/spark
      copying horovod/spark/gloo_run.py -> build/lib.linux-x86_64-cpython-311/horovod/spark
      copying horovod/spark/conf.py -> build/lib.linux-x86_64-cpython-311/horovod/spark
      copying horovod/spark/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/spark
      copying horovod/spark/runner.py -> build/lib.linux-x86_64-cpython-311/horovod/spark
      creating build/lib.linux-x86_64-cpython-311/horovod/mxnet
      copying horovod/mxnet/mpi_ops.py -> build/lib.linux-x86_64-cpython-311/horovod/mxnet
      copying horovod/mxnet/functions.py -> build/lib.linux-x86_64-cpython-311/horovod/mxnet
      copying horovod/mxnet/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/mxnet
      copying horovod/mxnet/compression.py -> build/lib.linux-x86_64-cpython-311/horovod/mxnet
      creating build/lib.linux-x86_64-cpython-311/horovod/tensorflow/keras
      copying horovod/tensorflow/keras/callbacks.py -> build/lib.linux-x86_64-cpython-311/horovod/tensorflow/keras
      copying horovod/tensorflow/keras/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/tensorflow/keras
      copying horovod/tensorflow/keras/elastic.py -> build/lib.linux-x86_64-cpython-311/horovod/tensorflow/keras
      creating build/lib.linux-x86_64-cpython-311/horovod/tensorflow/data
      copying horovod/tensorflow/data/compute_worker.py -> build/lib.linux-x86_64-cpython-311/horovod/tensorflow/data
      copying horovod/tensorflow/data/compute_service.py -> build/lib.linux-x86_64-cpython-311/horovod/tensorflow/data
      copying horovod/tensorflow/data/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/tensorflow/data
      creating build/lib.linux-x86_64-cpython-311/horovod/torch/elastic
      copying horovod/torch/elastic/sampler.py -> build/lib.linux-x86_64-cpython-311/horovod/torch/elastic
      copying horovod/torch/elastic/state.py -> build/lib.linux-x86_64-cpython-311/horovod/torch/elastic
      copying horovod/torch/elastic/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/torch/elastic
      creating build/lib.linux-x86_64-cpython-311/horovod/runner/http
      copying horovod/runner/http/http_server.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/http
      copying horovod/runner/http/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/http
      copying horovod/runner/http/http_client.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/http
      creating build/lib.linux-x86_64-cpython-311/horovod/runner/common
      copying horovod/runner/common/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/common
      creating build/lib.linux-x86_64-cpython-311/horovod/runner/util
      copying horovod/runner/util/remote.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/util
      copying horovod/runner/util/streams.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/util
      copying horovod/runner/util/cache.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/util
      copying horovod/runner/util/network.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/util
      copying horovod/runner/util/lsf.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/util
      copying horovod/runner/util/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/util
      copying horovod/runner/util/threads.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/util
      creating build/lib.linux-x86_64-cpython-311/horovod/runner/driver
      copying horovod/runner/driver/driver_service.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/driver
      copying horovod/runner/driver/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/driver
      creating build/lib.linux-x86_64-cpython-311/horovod/runner/task
      copying horovod/runner/task/task_service.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/task
      copying horovod/runner/task/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/task
      creating build/lib.linux-x86_64-cpython-311/horovod/runner/elastic
      copying horovod/runner/elastic/constants.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/elastic
      copying horovod/runner/elastic/settings.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/elastic
      copying horovod/runner/elastic/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/elastic
      copying horovod/runner/elastic/registration.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/elastic
      copying horovod/runner/elastic/driver.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/elastic
      copying horovod/runner/elastic/discovery.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/elastic
      copying horovod/runner/elastic/worker.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/elastic
      copying horovod/runner/elastic/rendezvous.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/elastic
      creating build/lib.linux-x86_64-cpython-311/horovod/runner/common/service
      copying horovod/runner/common/service/driver_service.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/common/service
      copying horovod/runner/common/service/compute_service.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/common/service
      copying horovod/runner/common/service/task_service.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/common/service
      copying horovod/runner/common/service/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/common/service
      creating build/lib.linux-x86_64-cpython-311/horovod/runner/common/util
      copying horovod/runner/common/util/tiny_shell_exec.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/common/util
      copying horovod/runner/common/util/config_parser.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/common/util
      copying horovod/runner/common/util/hosts.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/common/util
      copying horovod/runner/common/util/env.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/common/util
      copying horovod/runner/common/util/safe_shell_exec.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/common/util
      copying horovod/runner/common/util/settings.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/common/util
      copying horovod/runner/common/util/network.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/common/util
      copying horovod/runner/common/util/secret.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/common/util
      copying horovod/runner/common/util/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/common/util
      copying horovod/runner/common/util/timeout.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/common/util
      copying horovod/runner/common/util/host_hash.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/common/util
      copying horovod/runner/common/util/codec.py -> build/lib.linux-x86_64-cpython-311/horovod/runner/common/util
      creating build/lib.linux-x86_64-cpython-311/horovod/spark/common
      copying horovod/spark/common/params.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/common
      copying horovod/spark/common/_namedtuple_fix.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/common
      copying horovod/spark/common/constants.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/common
      copying horovod/spark/common/cache.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/common
      copying horovod/spark/common/store.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/common
      copying horovod/spark/common/estimator.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/common
      copying horovod/spark/common/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/common
      copying horovod/spark/common/backend.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/common
      copying horovod/spark/common/util.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/common
      copying horovod/spark/common/datamodule.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/common
      copying horovod/spark/common/serialization.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/common
      creating build/lib.linux-x86_64-cpython-311/horovod/spark/tensorflow
      copying horovod/spark/tensorflow/compute_worker.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/tensorflow
      copying horovod/spark/tensorflow/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/tensorflow
      creating build/lib.linux-x86_64-cpython-311/horovod/spark/keras
      copying horovod/spark/keras/optimizer.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/keras
      copying horovod/spark/keras/remote.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/keras
      copying horovod/spark/keras/bare.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/keras
      copying horovod/spark/keras/tensorflow.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/keras
      copying horovod/spark/keras/estimator.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/keras
      copying horovod/spark/keras/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/keras
      copying horovod/spark/keras/util.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/keras
      copying horovod/spark/keras/datamodule.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/keras
      creating build/lib.linux-x86_64-cpython-311/horovod/spark/torch
      copying horovod/spark/torch/remote.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/torch
      copying horovod/spark/torch/estimator.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/torch
      copying horovod/spark/torch/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/torch
      copying horovod/spark/torch/util.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/torch
      copying horovod/spark/torch/datamodule.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/torch
      creating build/lib.linux-x86_64-cpython-311/horovod/spark/driver
      copying horovod/spark/driver/host_discovery.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/driver
      copying horovod/spark/driver/driver_service.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/driver
      copying horovod/spark/driver/job_id.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/driver
      copying horovod/spark/driver/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/driver
      copying horovod/spark/driver/mpirun_rsh.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/driver
      copying horovod/spark/driver/rsh.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/driver
      copying horovod/spark/driver/rendezvous.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/driver
      creating build/lib.linux-x86_64-cpython-311/horovod/spark/task
      copying horovod/spark/task/gloo_exec_fn.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/task
      copying horovod/spark/task/mpirun_exec_fn.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/task
      copying horovod/spark/task/task_info.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/task
      copying horovod/spark/task/task_service.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/task
      copying horovod/spark/task/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/task
      creating build/lib.linux-x86_64-cpython-311/horovod/spark/data_loaders
      copying horovod/spark/data_loaders/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/data_loaders
      copying horovod/spark/data_loaders/pytorch_data_loaders.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/data_loaders
      creating build/lib.linux-x86_64-cpython-311/horovod/spark/lightning
      copying horovod/spark/lightning/remote.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/lightning
      copying horovod/spark/lightning/estimator.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/lightning
      copying horovod/spark/lightning/__init__.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/lightning
      copying horovod/spark/lightning/util.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/lightning
      copying horovod/spark/lightning/datamodule.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/lightning
      copying horovod/spark/lightning/legacy.py -> build/lib.linux-x86_64-cpython-311/horovod/spark/lightning
      running build_ext
      Traceback (most recent call last):
        File "/home/fschmidt/lib/test-venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/fschmidt/lib/test-venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/fschmidt/lib/test-venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-a7dm1qv3/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 434, in build_wheel
          return self._build_with_temp_dir(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-a7dm1qv3/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 419, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/pip-build-env-a7dm1qv3/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 507, in run_setup
          super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-a7dm1qv3/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 341, in run_setup
          exec(code, locals())
        File "<string>", line 213, in <module>
        File "/tmp/pip-build-env-a7dm1qv3/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-a7dm1qv3/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-a7dm1qv3/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-a7dm1qv3/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-a7dm1qv3/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 989, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-a7dm1qv3/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-a7dm1qv3/overlay/lib/python3.11/site-packages/wheel/bdist_wheel.py", line 369, in run
          self.run_command("build")
        File "/tmp/pip-build-env-a7dm1qv3/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-a7dm1qv3/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 989, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-a7dm1qv3/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-a7dm1qv3/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 131, in run
          self.run_command(cmd_name)
        File "/tmp/pip-build-env-a7dm1qv3/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-a7dm1qv3/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 989, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-a7dm1qv3/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-a7dm1qv3/overlay/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 88, in run
          _build_ext.run(self)
        File "/tmp/pip-build-env-a7dm1qv3/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
          self.build_extensions()
        File "<string>", line 106, in build_extensions
        File "<string>", line 67, in get_cmake_bin
      ModuleNotFoundError: No module named 'packaging'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for horovod
Failed to build horovod
ERROR: Could not build wheels for horovod, which is required to install pyproject.toml-based projects
@flixxox flixxox added the bug label Nov 10, 2023
@jklinkenberg
Copy link

Same issue here when trying to install Horovod on top of PyTorch 2.
Any fix for that yet?

@heliar-k
Copy link

heliar-k commented Jan 3, 2024

Same issue here when torch==2.1.2

@thangckt
Copy link

thangckt commented Jan 9, 2024

same issue with tensorflow=2.8.4

@EnricoMi EnricoMi linked a pull request Jan 9, 2024 that will close this issue
Open
@EnricoMi
Copy link
Collaborator

EnricoMi commented Jan 9, 2024

Can you try this fix please?

-HOROVOD_WITH_PYTORCH=1 pip3 install --no-cache-dir horovod[pytorch]
+HOROVOD_WITH_PYTORCH=1 pip3 install --no-cache-dir git+https://github.com/horovod/horovod.git@fix-packaging-not-found#egg=horovod[pytorch]

@thangckt
Copy link

I tried with this

HOROVOD_WITH_MPI=1 HOROVOD_WITH_TENSORFLOW=1 pip install -v --use-pep517 git+https://github.com/horovod/horovod.git@fix-packaging-not-found#egg=horovod

The problem "No module named 'packaging'" disappear, but there is the problem as below:

...
 ModuleNotFoundError: No module named 'tensorflow'
  CMake Error at /home1/p001cao/app/tooldev/cmake-3.27/share/cmake-3.27/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
    Could NOT find Tensorflow (missing: Tensorflow_LIBRARIES) (Required is at
    least version "1.15.0")
  Call Stack (most recent call first):
    /home1/p001cao/app/tooldev/cmake-3.27/share/cmake-3.27/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
    cmake/Modules/FindTensorflow.cmake:43 (find_package_handle_standard_args)
    horovod/tensorflow/CMakeLists.txt:12 (find_package)


  -- Configuring incomplete, errors occurred!
...

@EnricoMi
Copy link
Collaborator

... but there is the problem as below:

Did you install tensorflow before? Did you try

HOROVOD_WITH_MPI=1 HOROVOD_WITH_TENSORFLOW=1 pip install -v --use-pep517 git+https://github.com/horovod/horovod.git@fix-packaging-not-found#egg=horovod[tensorflow]

@thangckt
Copy link

dear @EnricoMi

I checked my tensorflow installation with

python -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"

and seemly that there is a problem

The TensorFlow library was compiled to use AVX instructions, but these aren't available on your machine.
Aborted

can you help with this?

@EnricoMi
Copy link
Collaborator

can you help with this?

Sorry, I can't. So that is unrelated and the branch might still work.

@EnricoMi
Copy link
Collaborator

EnricoMi commented Jan 10, 2024 •
edited

I can confirm the No module named 'packaging' can be reproduced with pip install --use-pep517. Fix #4016 fixes that bit, but --use-pep517 requires more fixes: #3991

For the time being, use pip install --use-pep517 --no-build-isolation horovod.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Do not skip entire require_list, only cffi when not build action horovod/horovod
5 participants
@jklinkenberg @EnricoMi @thangckt @flixxox @heliar-k