python3 setup.py install fails because FFTW3 doesn't support installation via CMake

[xiabingquan]

Bug Description

When creating python bindings for flashlight, we run python setup.py install in the directory bindings/python. The
scriptsetup.py starts a new sub-process:

flashlight/bindings/python/setup.py

Lines 99 to 104 in 2261fd9

	subprocess.check_call(
	["cmake", source_dir] + cmake_args, cwd=self.build_temp, env=env
	)
	subprocess.check_call(
	["cmake", "--build", "."] + build_args, cwd=self.build_temp
	)

It runs Cmake to build essential components. The problem is that FFTW3 doesn't support installation via CMake according to the reply in the issue:

However as the documentation says to use configure rather than cmake, the file FFTW3LibraryDepends.cmake may be missing on most installations. Is there any plan to make CMake the default compilation toolchain? Anyway, it would be worth supporting fftw detection through Cmake's find_package.

and the official documentation

Reproduction Steps

• cd ./repos/flashlight/bindings/python/
• Run command python ./setup.py install
• A trace of the error

running bdist_egg
running egg_info
writing flashlight.egg-info/PKG-INFO
writing dependency_links to flashlight.egg-info/dependency_links.txt
writing namespace_packages to flashlight.egg-info/namespace_packages.txt
writing top-level names to flashlight.egg-info/top_level.txt
reading manifest file 'flashlight.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'flashlight.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
running build_ext
-- -rdynamic supported.
-- CUDA found (library: /usr/local/cuda-11.1/lib64/libcudart_static.a;-lpthread;dl;/usr/lib/x86_64-linux-gnu/librt.so include: /usr/local/cuda-11.1/include)
-- OpenCL found (library: /usr/lib/x86_64-linux-gnu/libOpenCL.so include: /usr/include)
-- CUDA found (library: /usr/local/cuda-11.1/lib64/libcudart_static.a;-lpthread;dl;/usr/lib/x86_64-linux-gnu/librt.so include: /usr/local/cuda-11.1/include)
-- CUDA architecture flags: -gencodearch=compute_35,code=sm_35-gencodearch=compute_50,code=sm_50-gencodearch=compute_52,code=sm_52-gencodearch=compute_60,code=sm_60-gencodearch=compute_61,code=sm_61-gencodearch=compute_70,code=sm_70-gencodearch=compute_75,code=sm_75-gencodearch=compute_80,code=sm_80-gencodearch=compute_80,code=compute_80
-- MKL_THREADING = OMP
-- Checking for [mkl_intel_lp64 - mkl_gnu_thread - mkl_core - gomp - pthread - m - dl]
--   Library mkl_intel_lp64: /usr/lib/x86_64-linux-gnu/libmkl_intel_lp64.so
--   Library mkl_gnu_thread: /usr/lib/x86_64-linux-gnu/libmkl_gnu_thread.so
--   Library mkl_core: /usr/lib/x86_64-linux-gnu/libmkl_core.so
--   Library gomp: -fopenmp
--   Library pthread: /usr/lib/x86_64-linux-gnu/libpthread.so
--   Library m: /usr/lib/x86_64-linux-gnu/libm.so
--   Library dl: /usr/lib/x86_64-linux-gnu/libdl.so
-- MKL library found
-- CBLAS found (include: /usr/include/mkl, library: /usr/lib/x86_64-linux-gnu/libmkl_intel_lp64.so;/usr/lib/x86_64-linux-gnu/libmkl_gnu_thread.so;/usr/lib/x86_64-linux-gnu/libmkl_core.so;-fopenmp;/usr/lib/x86_64-linux-gnu/libpthread.so;/usr/lib/x86_64-linux-gnu/libm.so;/usr/lib/x86_64-linux-gnu/libdl.so)
CMake Error at /usr/local/lib/cmake/fftw3/FFTW3Config.cmake:14 (include):
  include could not find load file:

    /usr/local/lib/cmake/fftw3/FFTW3LibraryDepends.cmake
Call Stack (most recent call first):
  cmake/FindFFTW3.cmake:22 (find_package)
  flashlight/lib/audio/feature/CMakeLists.txt:17 (find_package)
  flashlight/lib/audio/CMakeLists.txt:16 (include)
  flashlight/lib/CMakeLists.txt:17 (include)
  CMakeLists.txt:118 (include)


-- FFTW found
-- Configuring incomplete, errors occurred!
See also "~/repos/flashlight/bindings/python/build/temp.linux-x86_64-cpython-38/CMakeFiles/CMakeOutput.log".
See also "~/repos/flashlight/bindings/python/build/temp.linux-x86_64-cpython-38/CMakeFiles/CMakeError.log".
Traceback (most recent call last):
  File "./setup.py", line 113, in <module>
    setup(
  File "~/miniconda3/envs/lrs/lib/python3.8/site-packages/setuptools/__init__.py", line 87, in setup
    return distutils.core.setup(**attrs)
  File "~/miniconda3/envs/lrs/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
  File "~/miniconda3/envs/lrs/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
  File ~/miniconda3/envs/lrs/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands
    self.run_command(cmd)
  File "~/miniconda3/envs/lrs/lib/python3.8/site-packages/setuptools/dist.py", line 1217, in run_command
    super().run_command(command)
  File "~/miniconda3/envs/lrs/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
    cmd_obj.run()
  File "~/miniconda3/envs/lrs/lib/python3.8/site-packages/setuptools/command/install.py", line 74, in run
    self.do_egg_install()
  File "~/miniconda3/envs/lrs/lib/python3.8/site-packages/setuptools/command/install.py", line 123, in do_egg_install
    self.run_command('bdist_egg')
  File "~/miniconda3/envs/lrs/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
    self.distribution.run_command(command)
  File "~/miniconda3/envs/lrs/lib/python3.8/site-packages/setuptools/dist.py", line 1217, in run_command
    super().run_command(command)
  File "~/miniconda3/envs/lrs/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
    cmd_obj.run()
  File "~/miniconda3/envs/lrs/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 165, in run
    cmd = self.call_command('install_lib', warn_dir=0)
  File "~/miniconda3/envs/lrs/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 151, in call_command
    self.run_command(cmdname)
  File "~/miniconda3/envs/lrs/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
    self.distribution.run_command(command)
  File "~/miniconda3/envs/lrs/lib/python3.8/site-packages/setuptools/dist.py", line 1217, in run_command
    super().run_command(command)
  File "~/miniconda3/envs/lrs/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
    cmd_obj.run()
  File "~/miniconda3/envs/lrs/lib/python3.8/site-packages/setuptools/command/install_lib.py", line 11, in run
    self.build()
  File "~/miniconda3/envs/lrs/lib/python3.8/site-packages/setuptools/_distutils/command/install_lib.py", line 112, in build
    self.run_command('build_ext')
  File "~/miniconda3/envs/lrs/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
    self.distribution.run_command(command)
  File "~/miniconda3/envs/lrs/lib/python3.8/site-packages/setuptools/dist.py", line 1217, in run_command
    super().run_command(command)
  File "~/miniconda3/envs/lrs/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
    cmd_obj.run()
  File "./setup.py", line 55, in run
    self.build_extensions(ext)
  File "./setup.py", line 99, in build_extensions
    subprocess.check_call(
  File "~/miniconda3/envs/lrs/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '~/repos/flashlight', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=~/repos/flashlight/bindings/python/build/lib.linux-x86_64-cpython-38', '-DPYTHON_EXECUTABLE=~/miniconda3/envs/lrs/bin/python3', '-DFL_BUILD_STANDALONE=OFF', '-DBUILD_SHARED_LIBS=ON', '-DFL_BUILD_CORE=OFF', '-DFL_BUILD_ALL_LIBS=ON', '-DFL_BUILD_EXAMPLES=OFF', '-DFL_BUILD_TESTS=OFF', '-DFL_LIBRARIES_BUILD_FOR_PYTHON=ON', '-DFL_LIBRARIES_USE_MKL=ON', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.

Platform and Hardware

• Linux avsu-ESC8000-G4 5.15.0-52-generic 58~20.04.1-Ubuntu SMP Thu Oct 13 13:09:46 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
• * GeForce RTX3090

Additional Context

No additional information

Solution

As we can see from the error message:

CMake Error at /usr/local/lib/cmake/fftw3/FFTW3Config.cmake:14 (include):
  include could not find load file:

    /usr/local/lib/cmake/fftw3/FFTW3LibraryDepends.cmake
Call Stack (most recent call first):
  cmake/FindFFTW3.cmake:22 (find_package)
  flashlight/lib/audio/feature/CMakeLists.txt:17 (find_package)
  flashlight/lib/audio/CMakeLists.txt:16 (include)
  flashlight/lib/CMakeLists.txt:17 (include)
  CMakeLists.txt:118 (include)


-- FFTW found
-- Configuring incomplete, errors occurred!

CMake could find the file FFTW3LibraryDepends.cmake due to the inherent bug of FFTW as mentioned before. However, we did have installed FFTW3 via ./configure && make && sudo make install. Since it shows -- FFTW found, we can safely ignore the error message and continue compiling the python bindings.

Solution: Change subprocess.check_call to subprocess.call to ignore CMake errors.

[maniyar1]

Just hit this, thank you for the information, you helped me figure out how to compile. Much appreciated.

EDIT: I spoke too soon x_x

Second EDIT: This did help me figure out how to compile! Just had a few other things to work through.

[xiabingquan]

Just hit this, thank you for the information, you helped me figure out how to compile. Much appreciated.

You're welcome. This bug did bother me for a while. Glad it helps.

[maniyar1]

In case people find this while searching, what I did in my Dockerfiles to fix this is I straight up deleted the fftw cmake directory:

RUN rm -rf /usr/local/lib/cmake/fftw3 # as per https://github.com/FFTW/fftw3/issues/130

This forces cmake to use pkg-config which worked for my case. Other options include compiling fftw using cmake, but because that wasn't documented well and has "missing features" I decided against it. This may be fixed for flashlight in #1060 but it is an upstream fftw bug as the original issue says.