AssertionError: MMCV==1.4.0 is used but incompatible. Please install mmcv>=2.0.0rc4, <2.2.0.

[DDEONSIK]

Prerequisite

• I have searched Issues and Discussions but cannot get the expected help.
• The bug has not been fixed in the latest version(https://github.com/open-mmlab/mmcv).

Environment

I understand that Ubuntu is installed in the 20.04 version.

python -c "from mmcv.utils import collect_env; print(collect_env())"

/bin/sh: 1: /usr/local/cuda/bin/nvcc: not found

{'sys.platform': 
'linux', 
'Python': '3.8.19 (default, Mar 20 2024, 19:58:24) [GCC 11.2.0]', 
'CUDA available': True, 
'GPU 0': 'NVIDIA A100-SXM4-80GB MIG 3g.40gb', 
'CUDA_HOME': '/usr/local/cuda', 
'NVCC': '', 
'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0', 
'PyTorch': '1.10.1', 
'PyTorch compiling details': 'PyTorch built with:\n  - GCC 7.3\n  - C++ Version: 201402\n  - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications\n  - Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)\n  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n  - LAPACK is enabled (usually provided by MKL)\n  - NNPACK is enabled\n  - CPU capability usage: AVX2\n  - CUDA Runtime 11.3\n  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37\n  - CuDNN 8.2\n  - Magma 2.5.2\n  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, 
CUDA_VERSION=11.3, 
CUDNN_VERSION=8.2.0, 
CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, 
CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, 
LAPACK_INFO=mkl, 
PERF_WITH_AVX=1, 
PERF_WITH_AVX2=1, 
PERF_WITH_AVX512=1, 
TORCH_VERSION=1.10.1, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, \n', 'TorchVision': '0.11.2', 
'OpenCV': '4.9.0', 
'MMCV': '1.4.0', 
'MMCV Compiler': 'GCC 7.3', 
'MMCV CUDA Compiler': '11.3'}

###############################################################

pip list

Package                   Version     Editable project location
------------------------- ----------- ------------------------------------------------
absl-py                   2.1.0
addict                    2.4.0
aiohttp                   3.9.3
aiosignal                 1.3.1
asttokens                 2.4.1
async-timeout             4.0.3
attrs                     23.2.0
backcall                  0.2.0
black                     24.3.0
blinker                   1.7.0
cachetools                5.3.3
casadi                    3.5.5
certifi                   2024.2.2
charset-normalizer        3.3.2
click                     8.1.7
comm                      0.2.2
ConfigArgParse            1.7
contourpy                 1.1.1
cycler                    0.12.1
dash                      2.16.1
dash-core-components      2.0.0
dash-html-components      2.0.0
dash-table                5.0.0
decorator                 5.1.1
descartes                 1.1.0
einops                    0.4.1
exceptiongroup            1.2.0
executing                 2.0.1
fastjsonschema            2.19.1
fire                      0.6.0
flake8                    7.0.0
Flask                     3.0.2
fonttools                 4.50.0
frozenlist                1.4.1
fsspec                    2024.3.1
future                    1.0.0
google-api-core           2.18.0
google-auth               2.29.0
google-auth-oauthlib      1.0.0
google-cloud-bigquery     3.19.0
google-cloud-core         2.4.1
google-crc32c             1.5.0
google-resumable-media    2.7.0
googleapis-common-protos  1.63.0
grpcio                    1.62.1
grpcio-status             1.62.1
idna                      3.6
imageio                   2.34.0
importlib_metadata        7.1.0
importlib_resources       6.4.0
iniconfig                 2.0.0
ipython                   8.12.3
ipywidgets                8.1.2
itsdangerous              2.1.2
jedi                      0.19.1
Jinja2                    3.1.3
joblib                    1.3.2
jsonschema                4.21.1
jsonschema-specifications 2023.12.1
jupyter_core              5.7.2
jupyterlab_widgets        3.0.10
kiwisolver                1.4.5
lazy_loader               0.3
lightning-utilities       0.11.0
llvmlite                  0.41.1
lyft-dataset-sdk          0.0.8
Markdown                  3.6
markdown-it-py            3.0.0
MarkupSafe                2.1.5
matplotlib                3.5.3
matplotlib-inline         0.1.6
mccabe                    0.7.0
mdurl                     0.1.2
mkl-fft                   1.3.1
mkl-random                1.2.2
mkl-service               2.4.0
mmcv-full                 1.4.0
mmdet                     2.14.0
mmdet3d                   1.4.0       /root/uuuuu/jjjjj/data/project/yyyyy/mmdetection3d
mmengine                  0.10.3
mmsegmentation            0.14.1
motmetrics                1.1.3
multidict                 6.0.5
mypy-extensions           1.0.0
nbformat                  5.10.3
nest-asyncio              1.6.0
networkx                  3.1
numba                     0.58.1
numpy                     1.24.3
nuscenes-devkit           1.1.11
oauthlib                  3.2.2
olefile                   0.47
open3d                    0.18.0
opencv-python             4.9.0.80
packaging                 24.0
pandas                    2.0.3
parso                     0.8.3
pathspec                  0.12.1
pexpect                   4.9.0
pickleshare               0.7.5
pillow                    10.2.0
pip                       23.3.1
pkgutil_resolve_name      1.3.10
platformdirs              4.2.0
plotly                    5.20.0
pluggy                    1.4.0
plyfile                   1.0.3
prettytable               3.10.0
prompt-toolkit            3.0.43
proto-plus                1.23.0
protobuf                  4.25.3
ptyprocess                0.7.0
pure-eval                 0.2.2
pyasn1                    0.5.1
pyasn1-modules            0.3.0
pycocotools               2.0.7
pycodestyle               2.11.1
pyflakes                  3.2.0
Pygments                  2.17.2
pyparsing                 3.1.2
pyquaternion              0.9.9
pytest                    8.1.1
python-dateutil           2.9.0.post0
pytorch-lightning         1.2.5
pytz                      2024.1
PyWavelets                1.4.1
PyYAML                    6.0.1
referencing               0.34.0
requests                  2.31.0
requests-oauthlib         1.4.0
retrying                  1.3.4
rich                      13.7.1
rpds-py                   0.18.0
rsa                       4.9
scikit-image              0.20.0
scikit-learn              1.3.2
scipy                     1.9.1
setuptools                68.2.2
Shapely                   1.8.5.post1
six                       1.16.0
stack-data                0.6.3
tenacity                  8.2.3
tensorboard               2.14.0
tensorboard-data-server   0.7.2
termcolor                 2.4.0
terminaltables            3.1.10
threadpoolctl             3.4.0
tifffile                  2023.7.10
tomli                     2.0.1
torch                     1.10.1
torchaudio                0.10.1
torchmetrics              1.3.2
torchvision               0.11.2
tqdm                      4.66.2
traitlets                 5.14.2
trimesh                   4.2.0
typing_extensions         4.10.0
tzdata                    2024.1
urllib3                   2.2.1
wcwidth                   0.2.13
Werkzeug                  3.0.1
wheel                     0.41.2
widgetsnbextension        4.0.10
yapf                      0.40.2
yarl                      1.9.4
zipp                      3.18.1

Reproduces the problem - code sample

nohup ./tools/uniad_dist_eval.sh ./projects/configs/stage1_track_map/base_track_map.py ./ckpts/uniad_base_track_map.pth 8 > output.log 2>&1 &

Reproduces the problem - command or script

nohup ./tools/uniad_dist_eval.sh ./projects/configs/stage1_track_map/base_track_map.py ./ckpts/uniad_base_track_map.pth 8 > output.log 2>&1 &

Reproduces the problem - error message

nohup: ignoring input
/root/anaconda3/envs/hhhhh/lib/python3.8/site-packages/torch/distributed/launch.py:178: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torchrun.
Note that --use_env is set by default in torchrun.
If your script expects `--local_rank` argument to be set, please
change it to read from `os.environ['LOCAL_RANK']` instead. See 
https://pytorch.org/docs/stable/distributed.html#launch-utility for 
further instructions

  warnings.warn(
WARNING:torch.distributed.run:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
Traceback (most recent call last):
  File "./tools/test.py", line 14, in <module>
    from mmdet3d.apis import single_gpu_test
  File "/root/uuuuu/jjjjj/data/project/yyyyy/mmdetection3d/mmdet3d/__init__.py", line 21, in <module>
    assert (mmcv_version >= digit_version(mmcv_minimum_version)
AssertionError: MMCV==1.4.0 is used but incompatible. Please install mmcv>=2.0.0rc4, <2.2.0.
Traceback (most recent call last):
  File "./tools/test.py", line 14, in <module>
    from mmdet3d.apis import single_gpu_test
  File "/root/uuuuu/jjjjj/data/project/yyyyy/mmdetection3d/mmdet3d/__init__.py", line 21, in <module>
    assert (mmcv_version >= digit_version(mmcv_minimum_version)
AssertionError: MMCV==1.4.0 is used but incompatible. Please install mmcv>=2.0.0rc4, <2.2.0.
Traceback (most recent call last):
  File "./tools/test.py", line 14, in <module>
    from mmdet3d.apis import single_gpu_test
  File "/root/uuuuu/jjjjj/data/project/yyyyy/mmdetection3d/mmdet3d/__init__.py", line 21, in <module>
    assert (mmcv_version >= digit_version(mmcv_minimum_version)
AssertionError: MMCV==1.4.0 is used but incompatible. Please install mmcv>=2.0.0rc4, <2.2.0.
Traceback (most recent call last):
  File "./tools/test.py", line 14, in <module>
    from mmdet3d.apis import single_gpu_test
  File "/root/uuuuu/jjjjj/data/project/yyyyy/mmdetection3d/mmdet3d/__init__.py", line 21, in <module>
    assert (mmcv_version >= digit_version(mmcv_minimum_version)
AssertionError: MMCV==1.4.0 is used but incompatible. Please install mmcv>=2.0.0rc4, <2.2.0.
Traceback (most recent call last):
  File "./tools/test.py", line 14, in <module>
    from mmdet3d.apis import single_gpu_test
  File "/root/uuuuu/jjjjj/data/project/yyyyy/mmdetection3d/mmdet3d/__init__.py", line 21, in <module>
    assert (mmcv_version >= digit_version(mmcv_minimum_version)
AssertionError: MMCV==1.4.0 is used but incompatible. Please install mmcv>=2.0.0rc4, <2.2.0.
Traceback (most recent call last):
  File "./tools/test.py", line 14, in <module>
    from mmdet3d.apis import single_gpu_test
  File "/root/uuuuu/jjjjj/data/project/yyyyy/mmdetection3d/mmdet3d/__init__.py", line 21, in <module>
    assert (mmcv_version >= digit_version(mmcv_minimum_version)
AssertionError: MMCV==1.4.0 is used but incompatible. Please install mmcv>=2.0.0rc4, <2.2.0.
Traceback (most recent call last):
  File "./tools/test.py", line 14, in <module>
    from mmdet3d.apis import single_gpu_test
  File "/root/uuuuu/jjjjj/data/project/yyyyy/mmdetection3d/mmdet3d/__init__.py", line 21, in <module>
    assert (mmcv_version >= digit_version(mmcv_minimum_version)
AssertionError: MMCV==1.4.0 is used but incompatible. Please install mmcv>=2.0.0rc4, <2.2.0.
Traceback (most recent call last):
  File "./tools/test.py", line 14, in <module>
    from mmdet3d.apis import single_gpu_test
  File "/root/uuuuu/jjjjj/data/project/yyyyy/mmdetection3d/mmdet3d/__init__.py", line 21, in <module>
    assert (mmcv_version >= digit_version(mmcv_minimum_version)
AssertionError: MMCV==1.4.0 is used but incompatible. Please install mmcv>=2.0.0rc4, <2.2.0.
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 507441) of binary: /root/anaconda3/envs/hhhhh/bin/python
Traceback (most recent call last):
  File "/root/anaconda3/envs/hhhhh/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/root/anaconda3/envs/hhhhh/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/root/anaconda3/envs/hhhhh/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in <module>
    main()
  File "/root/anaconda3/envs/hhhhh/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main
    launch(args)
  File "/root/anaconda3/envs/hhhhh/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch
    run(args)
  File "/root/anaconda3/envs/hhhhh/lib/python3.8/site-packages/torch/distributed/run.py", line 710, in run
    elastic_launch(
  File "/root/anaconda3/envs/hhhhh/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/root/anaconda3/envs/hhhhh/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 259, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
./tools/test.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2024-03
  host      : uuuuu00000
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 00000)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2024-03
  host      : uuuuu00000
  rank      : 2 (local_rank: 2)
  exitcode  : 1 (pid: 00000)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
  time      : 2024-03
  host      : uuuuu00000
  rank      : 3 (local_rank: 3)
  exitcode  : 1 (pid: 00000)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[4]:
  time      : 2024-03
  host      : uuuuu00000
  rank      : 4 (local_rank: 4)
  exitcode  : 1 (pid: 507449)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[5]:
  time      : 2024-03
  host      : uuuuu00000
  rank      : 5 (local_rank: 5)
  exitcode  : 1 (pid: 00000)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[6]:
  time      : 2024-03
  host      : uuuuu00000
  rank      : 6 (local_rank: 6)
  exitcode  : 1 (pid: 00000)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[7]:
  time      : 2024-03
  host      : uuuuu00000
  rank      : 7 (local_rank: 7)
  exitcode  : 1 (pid: 00000)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-03
  host      : uuuuu00000
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 00000)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

Ubuntu Ver: 20.04 / CUDA 11.3 / cudatoolkit=11.3 / Torch=1.10.1 / Python 3.8.19

Additional information

I'm going to try UniAD.

Annotation processing for error resolution

.. /mmdetection3d/requirements/runtime.txt

lyft_dataset_sdk
# networkx>=2.5
numba # you should install numba==0.53.0 if your environment is cuda-9.0
# numpy
nuscenes-devkit
open3d
plyfile
scikit-image
# by default we also use tensorboard to log results
tensorboard
trimesh

###############################################################

Process

env setting -- hhhhh

conda create -n hhhhh python=3.8 -y
conda env list
conda activate hhhhh
conda list

cd root/uuuuu/jjjjj/data/project/yyyyy
ctrl+shift+p : >select interpreter : hhhhh

Torch install - cudatoolkit=11.3 / Torch=1.10.1

conda install pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=11.3 -c pytorch -c conda-forge -y 
conda list cudatoolkit -f 
python -c "import torch; print(torch.__version__)"

export PATH=/usr/local/cuda-11.3/bin:$PATH 

export CUDA_HOME=/mnt/cuda-11.3/ 

install mmcv

pip install mmcv-full==1.4.0 -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.10.0/index.html
pip show mmcv-full 

install mmdet, mmseg

pip install mmdet==2.14.0
pip show mmdet

pip install mmsegmentation==0.14.1
pip show mmsegmentation

install mmdet3d

git clone https://github.com/open-mmlab/mmdetection3d.git
cd mmdetection3d
git checkout v1.4.0 #git:Refresh
pip install scipy==1.7.3
pip show scipy 
/root/uuuuu/jjjjj/data/project/yyyyy/mmdetection3d/requirements/runtime.txt
pip install scikit-image==0.20.0  
pip uninstall networkx
pip show scikit-image 
pip install -v -e .

install UniAD

git clone https://github.com/OpenDriveLab/UniAD.git
cd UniAD
pip install -r requirements.txt 
pip install numpy==1.24.3
pip install numpy==1.20.0
pip show numpy 

install pretrained weights

mkdir ckpts && cd ckpts
wget https://github.com/zhiqi-li/storage/releases/download/v1.0/bevformer_r101_dcn_24ep.pth
wget https://github.com/OpenDriveLab/UniAD/releases/download/v1.0/uniad_base_track_map.pth
wget https://github.com/OpenDriveLab/UniAD/releases/download/v1.0.1/uniad_base_e2e.pth
cd ..

nuScenes data

cd UniAD
mkdir data && cd data
ln -s /root/uuuuu/jjjjj/data/dataset/nuscenes /root/uuuuu/jjjjj/data/project/yyyyy/UniAD/data #DATA Symbolic Link

mkdir infos && cd infos
wget https://github.com/OpenDriveLab/UniAD/releases/download/v1.0/nuscenes_infos_temporal_train.pkl
wget https://github.com/OpenDriveLab/UniAD/releases/download/v1.0/nuscenes_infos_temporal_val.pkl

cd UniAD/data
cd ..
mkdir others && cd others
wget https://github.com/OpenDriveLab/UniAD/releases/download/v1.0/motion_anchor_infos_mode6.pkl

Evaluation Example

cd UniAD
nohup ./tools/uniad_dist_eval.sh ./projects/configs/stage1_track_map/base_track_map.py ./ckpts/uniad_base_track_map.pth 8 > output.log 2>&1 &

nohup ./tools/uniad_dist_eval.sh ./projects/configs/stage2_e2e/base_e2e.py ./ckpts/uniad_base_e2e.pth 8 > output.log 2>&1 &

Please solve the MMCV version problem, thank you.

I am also considering the issue of the giteckout version.

[DDEONSIK]

Problem Cause: The git checkout version of mmdet3d was not matched.
Thought I had to match the gitcheckout version for CUDA 11.3 and Torch 1.10.1. But it was related to mmcv.
Workaround: I set the git checkout version to 0.17.1, rerouted it, and reinstalled the files.

cd ~
git clone https://github.com/open-mmlab/mmdetection3d.git
cd mmdetection3d
git checkout v0.17.1
pip install scipy==1.7.3
pip install scikit-image==0.20.0
pip install -v -e .

I also processed the code below in the "../mmdetection3d/requirements/runtime.txt" path.

#networkx>=2.2,<2.3
#numpy<1.20.0

In subsequent execution, it is estimated that there will be a problem with the input data.
Estimated to add a few lines of code to the path.
.. /UniAD/projects/stage1_track_map/base_track_map.py".