Add support for Hydra MPI #2761

dzhwinter · 2020-10-21T13:34:56Z

dzhwinter
Oct 21, 2020

2020-10-21 21:17:16.484873: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Unknown MPI implementation given in output of mpirun --version:
HYDRA build details:
    Version:                                 3.3.2
    Release Date:                            Tue Nov 12 21:23:16 CST 2019
    CC:                              gcc
    CXX:                             g++
    F77:                             gfortran
    F90:                             gfortran
    Configure options:                       '--disable-option-checking' '--prefix=/usr/local' '--cache-file=/dev/null' '--srcdir=.' 'CC=gcc' 'CFLAGS= -O2' 'LDFLAGS=' 'LIBS=' 'CPPFLAGS= -I/root/mpich-3.3.2/src/mpl/include -I/root/mpich-3.3.2/src/mpl/include -I/root/mpich-3.3.2/src/openpa/src -I/root/mpich-3.3.2/src/openpa/src -D_REENTRANT -I/root/mpich-3.3.2/src/mpi/romio/include' 'MPLLIBNAME=mpl'
    Process Manager:                         pmi
    Launchers available:                     ssh rsh fork slurm ll lsf sge manual persist
    Topology libraries available:            hwloc
    Resource management kernels available:   user slurm ll lsf sge pbs cobalt
    Checkpointing libraries available:
    Demux engines available:                 poll select

Unknown MPI implementation given in output of mpirun --version:
HYDRA build details:
    Version:                                 3.3.2
    Release Date:                            Tue Nov 12 21:23:16 CST 2019
    CC:                              gcc
    CXX:                             g++
    F77:                             gfortran
    F90:                             gfortran
    Configure options:                       '--disable-option-checking' '--prefix=/usr/local' '--cache-file=/dev/null' '--srcdir=.' 'CC=gcc' 'CFLAGS= -O2' 'LDFLAGS=' 'LIBS=' 'CPPFLAGS= -I/root/mpich-3.3.2/src/mpl/include -I/root/mpich-3.3.2/src/mpl/include -I/root/mpich-3.3.2/src/openpa/src -I/root/mpich-3.3.2/src/openpa/src -D_REENTRANT -I/root/mpich-3.3.2/src/mpi/romio/include' 'MPLLIBNAME=mpl'
    Process Manager:                         pmi
    Launchers available:                     ssh rsh fork slurm ll lsf sge manual persist
    Topology libraries available:            hwloc
    Resource management kernels available:   user slurm ll lsf sge pbs cobalt
    Checkpointing libraries available:
    Demux engines available:                 poll select

Unknown MPI implementation given in output of mpirun --version:
HYDRA build details:
    Version:                                 3.3.2
    Release Date:                            Tue Nov 12 21:23:16 CST 2019
    CC:                              gcc
    CXX:                             g++
    F77:                             gfortran
    F90:                             gfortran
    Configure options:                       '--disable-option-checking' '--prefix=/usr/local' '--cache-file=/dev/null' '--srcdir=.' 'CC=gcc' 'CFLAGS= -O2' 'LDFLAGS=' 'LIBS=' 'CPPFLAGS= -I/root/mpich-3.3.2/src/mpl/include -I/root/mpich-3.3.2/src/mpl/include -I/root/mpich-3.3.2/src/openpa/src -I/root/mpich-3.3.2/src/openpa/src -D_REENTRANT -I/root/mpich-3.3.2/src/mpi/romio/include' 'MPLLIBNAME=mpl'
    Process Manager:                         pmi
    Launchers available:                     ssh rsh fork slurm ll lsf sge manual persist
    Topology libraries available:            hwloc
    Resource management kernels available:   user slurm ll lsf sge pbs cobalt
    Checkpointing libraries available:
    Demux engines available:                 poll select

Traceback (most recent call last):
  File "/dockerdata/anaconda3/bin/horovodrun", line 10, in <module>
    sys.exit(run_commandline())
  File "/dockerdata/anaconda3/lib/python3.7/site-packages/horovod/runner/launch.py", line 723, in run_commandline
    _run(args)
  File "/dockerdata/anaconda3/lib/python3.7/site-packages/horovod/runner/launch.py", line 713, in _run
    return _run_static(args)
  File "/dockerdata/anaconda3/lib/python3.7/site-packages/horovod/runner/launch.py", line 571, in _run_static
    _launch_job(args, settings, nics, command)
  File "/dockerdata/anaconda3/lib/python3.7/site-packages/horovod/runner/launch.py", line 686, in _launch_job
    args.verbose)
  File "/dockerdata/anaconda3/lib/python3.7/site-packages/horovod/runner/launch.py", line 657, in run_controller
    mpi_run()
  File "/dockerdata/anaconda3/lib/python3.7/site-packages/horovod/runner/launch.py", line 678, in mpi_run_fn
    mpi_run(settings, nics, env, command)
  File "/dockerdata/anaconda3/lib/python3.7/site-packages/horovod/runner/mpi_run.py", line 143, in mpi_run
    raise Exception(_MPI_NOT_FOUND_ERROR_MSG)
Exception: horovod does not find an installed MPI.

Choose one of:
1. Install Open MPI 4.0.0+ or IBM Spectrum MPI or MPICH and re-install Horovod (use --no-cache-dir pip option).
2. Run distributed training script using the standard way provided by your MPI distribution (usually mpirun, srun, or jsrun).
3. Use built-in gloo option (horovodrun --gloo ...).

The check-build command output

Checking whether extension tensorflow was built.
2020-10-21 21:28:31.314100: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Extension tensorflow was built.
Checking whether extension torch was built.
Traceback (most recent call last):
  File "/dockerdata/anaconda3/lib/python3.7/site-packages/horovod/torch/__init__.py", line 21, in <module>
    __file__, 'mpi_lib_v2')
  File "/dockerdata/anaconda3/lib/python3.7/site-packages/horovod/common/util.py", line 56, in check_extension
    ext_name, full_path, ext_env_var
ImportError: Extension horovod.torch has not been built: /dockerdata/anaconda3/lib/python3.7/site-packages/horovod/torch/mpi_lib_v2.cpython-37m-x86_64-linux-gnu.so not found
If this is not expected, reinstall Horovod with HOROVOD_WITH_PYTORCH=1 to debug the build error.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/dockerdata/anaconda3/lib/python3.7/site-packages/horovod/common/util.py", line 80, in _target_fn
    ext = importlib.import_module('.' + ext_base_name, 'horovod')
  File "/dockerdata/anaconda3/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/dockerdata/anaconda3/lib/python3.7/site-packages/horovod/torch/__init__.py", line 24, in <module>
    __file__, 'mpi_lib', '_mpi_lib')
  File "/dockerdata/anaconda3/lib/python3.7/site-packages/horovod/common/util.py", line 56, in check_extension
    ext_name, full_path, ext_env_var
ImportError: Extension horovod.torch has not been built: /dockerdata/anaconda3/lib/python3.7/site-packages/horovod/torch/mpi_lib/_mpi_lib.cpython-37m-x86_64-linux-gnu.so not found
If this is not expected, reinstall Horovod with HOROVOD_WITH_PYTORCH=1 to debug the build error.
Extension torch was NOT built.
Checking whether extension mxnet was built.
Traceback (most recent call last):
  File "/dockerdata/anaconda3/lib/python3.7/site-packages/horovod/common/util.py", line 80, in _target_fn
    ext = importlib.import_module('.' + ext_base_name, 'horovod')
  File "/dockerdata/anaconda3/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/dockerdata/anaconda3/lib/python3.7/site-packages/horovod/mxnet/__init__.py", line 19, in <module>
    __file__, 'mpi_lib')
  File "/dockerdata/anaconda3/lib/python3.7/site-packages/horovod/common/util.py", line 56, in check_extension
    ext_name, full_path, ext_env_var
ImportError: Extension horovod.mxnet has not been built: /dockerdata/anaconda3/lib/python3.7/site-packages/horovod/mxnet/mpi_lib.cpython-37m-x86_64-linux-gnu.so not found
If this is not expected, reinstall Horovod with HOROVOD_WITH_MXNET=1 to debug the build error.
Extension mxnet was NOT built.
Checking whether extension tensorflow was built with MPI.
2020-10-21 21:28:33.215982: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Extension tensorflow was built with MPI.
Checking whether extension tensorflow was built with Gloo.
2020-10-21 21:28:35.069995: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Extension tensorflow was built with Gloo.
Checking whether extension tensorflow was built with NCCL.
2020-10-21 21:28:36.950945: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Extension tensorflow was built with NCCL.
Checking whether extension tensorflow was built with DDL.
2020-10-21 21:28:38.789250: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Extension tensorflow was NOT built with DDL.
Checking whether extension tensorflow was built with CCL.
2020-10-21 21:28:40.626667: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Extension tensorflow was NOT built with CCL.

Horovod v0.20.3:

Available Frameworks:
    [X] TensorFlow
    [ ] PyTorch
    [ ] MXNet

Available Controllers:
    [X] MPI
    [X] Gloo

Available Tensor Operations:
    [X] NCCL
    [ ] DDL
    [ ] CCL
    [X] MPI
    [X] Gloo

tgaddair · 2020-10-21T14:58:32Z

tgaddair
Oct 21, 2020
Maintainer

Hey @dzhwinter, can you run mpirun --version and share the output here?

Looks like our parser is not detecting your version correctly.

0 replies

tgaddair · 2020-10-21T15:08:47Z

tgaddair
Oct 21, 2020
Maintainer

Nevermind, should have looked closer at the output above.

0 replies

tgaddair · 2020-10-21T15:16:18Z

tgaddair
Oct 21, 2020
Maintainer

@dzhwinter, the issue here is that we look for the string "MPICH" in the output of mpirun --version to detect an MPICH version of MPI. In this case, we see HYDRA where we would expect to see MPICH.

Do you know if the Hydra version of mpirun supports the same set of arguments as MPICH mpirun? If so, we can change the parser to treat them the same. If not, I would suggest calling mpirun directly for now until we have support for the different argument set.

cc @nvcastet in case you happen to be familiar with this version of MPI.

0 replies

nvcastet · 2020-10-23T12:42:11Z

nvcastet
Oct 23, 2020
Collaborator

@dzhwinter Do you know if the Hydra-based mpirun command takes the same arguments as the stock MPICH mpirun command?
You could try modifying https://github.com/horovod/horovod/blob/master/horovod/runner/mpi_run.py#L99 with:
elif 'MPICH' in output or 'HYDRA' in output:
If that works we could add a PR for this change.

0 replies

baskrahmer · 2020-10-23T15:10:42Z

baskrahmer
Oct 23, 2020

I stumbled upon the same error. Simply switching from horovodrun to mpirun as @tgaddair suggested gets the job done. Furthermore, I think the two versions take the same arguments. Below is the output of mpirun --help for the Hydra build, maybe it's useful for comparison.

$ mpirun --help

Usage: ./mpiexec [global opts] [local opts for exec1] [exec1] [exec1 args] : [local opts for exec2] [exec2] [exec2 args] : ...

Global options (passed to all executables):

  Global environment options:
    -genv {name} {value}             environment variable name and value
    -genvlist {env1,env2,...}        environment variable list to pass
    -genvnone                        do not pass any environment variables
    -genvall                         pass all environment variables not managed
                                          by the launcher (default)

  Other global options:
    -f {name}                        file containing the host names
    -hosts {host list}               comma separated host list
    -wdir {dirname}                  working directory to use
    -configfile {name}               config file containing MPMD launch options


Local options (passed to individual executables):

  Local environment options:
    -env {name} {value}              environment variable name and value
    -envlist {env1,env2,...}         environment variable list to pass
    -envnone                         do not pass any environment variables
    -envall                          pass all environment variables (default)

  Other local options:
    -n/-np {value}                   number of processes
    {exec_name} {args}               executable name and arguments


Hydra specific options (treated as global):

  Launch options:
    -launcher                        launcher to use (ssh rsh fork slurm ll lsf sge manual persist)
    -launcher-exec                   executable to use to launch processes
    -enable-x/-disable-x             enable or disable X forwarding

  Resource management kernel options:
    -rmk                             resource management kernel to use (user slurm ll lsf sge pbs cobalt)

  Processor topology options:
    -topolib                         processor topology library (hwloc)
    -bind-to                         process binding
    -map-by                          process mapping
    -membind                         memory binding policy

  Checkpoint/Restart options:
    -ckpoint-interval                checkpoint interval
    -ckpoint-prefix                  checkpoint file prefix
    -ckpoint-num                     checkpoint number to restart
    -ckpointlib                      checkpointing library (none)

  Demux engine options:
    -demux                           demux engine (poll select)

  Other Hydra options:
    -verbose                         verbose mode
    -info                            build information
    -print-all-exitcodes             print exit codes of all processes
    -iface                           network interface to use
    -ppn                             processes per node
    -profile                         turn on internal profiling
    -prepend-rank                    prepend rank to output
    -prepend-pattern                 prepend pattern to output
    -outfile-pattern                 direct stdout to file
    -errfile-pattern                 direct stderr to file
    -nameserver                      name server information (host:port format)
    -disable-auto-cleanup            don't cleanup processes on error
    -disable-hostname-propagation    let MPICH auto-detect the hostname
    -order-nodes                     order nodes as ascending/descending cores
    -localhost                       local hostname for the launching node
    -usize                           universe size (SYSTEM, INFINITE, <value>)

Please see the intructions provided at
http://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager
for further details

0 replies

dzhwinter · 2020-10-30T16:30:07Z

dzhwinter
Oct 30, 2020
Author

@tgaddair @nvcastet HYDRA is a brand new process manager of MPICH, the default configuration. It takes the same arguments of the naked mpirun.

Check the details in
https://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager

0 replies

tgaddair · 2020-10-30T17:59:00Z

tgaddair
Oct 30, 2020
Maintainer

Thanks @dzhwinter, in the case we should be able to add it as a supported MPI version in horovodrun using the same config as MPICH.

0 replies

dzhwinter · 2020-11-01T16:07:42Z

dzhwinter
Nov 1, 2020
Author

I think we should add the new MPICH in the CI. Given that HYDRA is the default configuration of the future.

0 replies

tgaddair · 2020-11-01T22:00:23Z

tgaddair
Nov 1, 2020
Maintainer

That sounds good to me, feel free to submit a PR to add HYDRA to the CI if you have the time.

0 replies

njzjz · 2023-09-12T21:46:02Z

njzjz
Sep 12, 2023

Hi, has this problem been resolved? I still get this problem with the latest version.

Unknown MPI implementation given in output of mpirun --version:
HYDRA build details:
    Version:                                 4.0
    Release Date:                            Fri Jan 21 10:42:29 CST 2022
    CC:                              gcc -Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -ffile-prefix-map=/build/mpich-0xgrG5/mpich-4.0=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security  -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro
    Configure options:                       '--with-hwloc-prefix=/usr' '--with-device=ch4:ofi' 'FFLAGS=-O2 -ffile-prefix-map=/build/mpich-0xgrG5/mpich-4.0=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -fallow-invalid-boz -fallow-argument-mismatch' '--prefix=/usr' 'CFLAGS=-g -O2 -ffile-prefix-map=/build/mpich-0xgrG5/mpich-4.0=. -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security' 'LDFLAGS=-Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2'
    Process Manager:                         pmi
    Launchers available:                     ssh rsh fork slurm ll lsf sge manual persist
    Topology libraries available:            hwloc
    Resource management kernels available:   user slurm ll lsf sge pbs cobalt
    Demux engines available:                 poll select

1 reply

Patataman Feb 7, 2024

It's fixed https://github.com/horovod/horovod/blob/master/horovod/runner/mpi_run.py#L99, yet the v0.28.1 still use the previous version

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Hydra MPI #2761

{{title}}

Replies: 10 comments 1 reply

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Add support for Hydra MPI #2761

dzhwinter Oct 21, 2020

Replies: 10 comments · 1 reply

tgaddair Oct 21, 2020 Maintainer

tgaddair Oct 21, 2020 Maintainer

tgaddair Oct 21, 2020 Maintainer

nvcastet Oct 23, 2020 Collaborator

baskrahmer Oct 23, 2020

dzhwinter Oct 30, 2020 Author

tgaddair Oct 30, 2020 Maintainer

dzhwinter Nov 1, 2020 Author

tgaddair Nov 1, 2020 Maintainer

njzjz Sep 12, 2023

Patataman Feb 7, 2024

dzhwinter
Oct 21, 2020

Replies: 10 comments 1 reply

tgaddair
Oct 21, 2020
Maintainer

tgaddair
Oct 21, 2020
Maintainer

tgaddair
Oct 21, 2020
Maintainer

nvcastet
Oct 23, 2020
Collaborator

baskrahmer
Oct 23, 2020

dzhwinter
Oct 30, 2020
Author

tgaddair
Oct 30, 2020
Maintainer

dzhwinter
Nov 1, 2020
Author

tgaddair
Nov 1, 2020
Maintainer

njzjz
Sep 12, 2023