Real-time mode issue: Problem encountered when trying to connect ZED2 camera

[TonyDua]

Hello, muskie82,x

First of all, thank you and your team very much for your hard work. Secondly, I am currently testing the use of the live mode. I noticed that you are using the Intel RealSense D455, but we do not have this device, we only have the ZED2, which also has dual RGB and depth cameras. I noticed that you defined the RealsenseDataset class in utils\dataset.py, where the init method disables auto exposure, sets custom exposure to 200, then obtains the camera intrinsics and undistorts the images. Finally, in the getitem method, the camera images are obtained, converted to torch tensors via cv2 color mode conversion, and then returned along with the pose.

Therefore, following the example of your code, we have also defined a class for the ZED (which also provides a Python package), the code is as follows.

class ZedDataset(BaseDataset):
    def __init__(self, args, path, config):
        super().__init__(args, path, config)
        self.zed = sl.Camera()
        init_params = sl.InitParameters()
        init_params.camera_resolution = sl.RESOLUTION.HD720
        init_params.camera_fps = 30
        init_params.depth_mode = sl.DEPTH_MODE.NONE
        # Open the camera
        err = self.zed.open(init_params)
        if err != sl.ERROR_CODE.SUCCESS:
            print("Camera Open : "+repr(err)+". Exit program.")
            exit()
        self.zed_cam_info = self.zed.get_camera_information()
        self.zed_calibration_params = self.zed_cam_info.camera_configuration.calibration_parameters
         #  left eye 
        self.fx = self.zed_calibration_params.left_cam.fx
        self.fy = self.zed_calibration_params.left_cam.fy
        self.cx = self.zed_calibration_params.left_cam.cx
        self.cy = self.zed_calibration_params.left_cam.cy
        self.width = self.zed_calibration_params.left_cam.image_size.width
        self.height = self.zed_calibration_params.left_cam.image_size.height
        self.fovx = self.zed_calibration_params.left_cam.v_fov
        self.fovy = self.zed_calibration_params.left_cam.h_fov
        self.K = np.array(
            [[self.fx, 0.0, self.cx], [0.0, self.fy, self.cy], [0.0, 0.0, 1.0]]
        )
        
        self.zed_runtime = sl.RuntimeParameters()

    def __getitem__(self, idx):
        pose = torch.eye(4, device=self.device, dtype=self.dtype)
        try : 
            err = self.zed.grab(self.zed_runtime)
            mat = sl.Mat() 
            if err == sl.ERROR_CODE.SUCCESS: 
                # Retrieve left image
                self.zed.retrieve_image(mat, sl.VIEW.LEFT)
                cvImage = mat.get_data() # Convert sl.Mat to cv2.Mat
                #image = np.asanyarray(rgb_frame.get_data())
                frame = cv2.cvtColor(cvImage, cv2.COLOR_BGR2RGB)
                
                image = (
                    torch.from_numpy(frame / 255.0)
                    .clamp(0.0, 1.0)
                    .permute(2, 0, 1)
                    .to(device=self.device, dtype=self.dtype)
                )
                return image, None, pose
        except KeyboardInterrupt :
            exit()
        except Exception as e:
            print(e)
            exit()

And in the load_dataset method, add a check to read the ZED configuration file.

def load_dataset(args, path, config):
    if config["Dataset"]["type"] == "tum":
        return TUMDataset(args, path, config)
    elif config["Dataset"]["type"] == "replica":
        return ReplicaDataset(args, path, config)
    elif config["Dataset"]["type"] == "euroc":
        return EurocDataset(args, path, config)
    elif config["Dataset"]["type"] == "realsense":
        return RealsenseDataset(args, path, config)
    # Zed
    elif config["Dataset"]["type"] == "zed":
        return ZedDataset(args, path, config)
    else:
        raise ValueError("Unknown dataset type")

So far, we have encountered many issues, including the projection matrix calculation part, but we have debugged them all and found no problems during testing. Before starting your project, we confirmed the normal operation of the ZED2 camera through OpenCV and the official demo program provided by ZED.

However, after starting the live mode, although we can see the image from the ZED2 camera in the viewport, the frame rate of the camera preview is low. Additionally, there are no Gaussian calculation results in the viewport. When we slightly move the camera, the program crashes with the following error message, and then it freezes.

[2024-04-29 04:06:57 UTC][ZED][INFO] Logging level INFO
[2024-04-29 04:06:58 UTC][ZED][INFO] [Init]  Depth mode: NONE
[2024-04-29 04:06:59 UTC][ZED][INFO] [Init]  Camera successfully opened.
[2024-04-29 04:06:59 UTC][ZED][INFO] [Init]  Camera FW version: 1523
[2024-04-29 04:06:59 UTC][ZED][INFO] [Init]  Video mode: HD720@30
[2024-04-29 04:06:59 UTC][ZED][INFO] [Init]  Serial Number: S/N 22103256
FEngine (64 bits) created at 000002AF9F29E050 (threading is enabled)
FEngine resolved backend: OpenGL
MonoGS: Resetting the system
MonoGS: Initialized map
Process Process-4:
Traceback (most recent call last):
  File "C:\ProgramData\anaconda3\envs\MonoGS\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
  File "C:\ProgramData\anaconda3\envs\MonoGS\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "C:\AI_Workstation\MonoGS\gui\slam_gui.py", line 688, in run
    app.run()
  File "C:\AI_Workstation\MonoGS\gui\slam_gui.py", line 676, in update
    self.scene_update()
  File "C:\AI_Workstation\MonoGS\gui\slam_gui.py", line 662, in scene_update
    self.receive_data(self.q_main2vis)
  File "C:\AI_Workstation\MonoGS\gui\slam_gui.py", line 406, in receive_data
    frustum = self.add_camera(
  File "C:\AI_Workstation\MonoGS\gui\slam_gui.py", line 256, in add_camera
    else getWorld2View2(camera.R, camera.T)
  File "C:\AI_Workstation\MonoGS\gaussian_splatting\utils\graphics_utils.py", line 41, in getWorld2View2
    C2W = torch.linalg.inv(Rt)
torch._C._LinAlgError: linalg.inv: The diagonal element 1 is zero, the inversion could not be completed because the input matrix is singular.
Traceback (most recent call last):
  File "C:\AI_Workstation\MonoGS\slam.py", line 252, in <module>
    slam = SLAM(config, save_dir=save_dir)
  File "C:\AI_Workstation\MonoGS\slam.py", line 110, in __init__
    self.frontend.run()
  File "C:\AI_Workstation\MonoGS\utils\slam_frontend.py", line 392, in run
    render_pkg = self.tracking(cur_frame_idx, viewpoint)
  File "C:\AI_Workstation\MonoGS\utils\slam_frontend.py", line 164, in tracking
    render_pkg = render(
  File "C:\AI_Workstation\MonoGS\gaussian_splatting\gaussian_renderer\__init__.py", line 65, in render
    viewmatrix=viewpoint_camera.world_view_transform,
  File "C:\AI_Workstation\MonoGS\utils\camera_utils.py", line 96, in world_view_transform
    return getWorld2View2(self.R, self.T).transpose(0, 1)
  File "C:\AI_Workstation\MonoGS\gaussian_splatting\utils\graphics_utils.py", line 41, in getWorld2View2
    C2W = torch.linalg.inv(Rt)
torch._C._LinAlgError: linalg.inv: The diagonal element 1 is zero, the inversion could not be completed because the input matrix is singular.

We are looking forward to your assistance. Thank you!

Operating environment: Windows 11
CUDA: 11.8
PyTorch version installed is compatible.

[TonyDua]

# conda list
# packages in environment at C:\ProgramData\anaconda3\envs\MonoGS:
#
# Name                    Version                   Build  Channel
appdirs                   1.4.4                    pypi_0    pypi
argcomplete               3.3.0                    pypi_0    pypi
asttokens                 2.4.1                    pypi_0    pypi
attrs                     23.2.0                   pypi_0    pypi
blas                      1.0                         mkl
blinker                   1.7.0                    pypi_0    pypi
brotli-python             1.0.9            py39hd77b12b_7
ca-certificates           2024.3.11            haa95532_0
certifi                   2024.2.2           pyhd8ed1ab_0    conda-forge
charset-normalizer        2.0.4              pyhd3eb1b0_0
click                     8.1.7                    pypi_0    pypi
colorama                  0.4.6            py39haa95532_0
comm                      0.2.2                    pypi_0    pypi
configargparse            1.7                      pypi_0    pypi
contourpy                 1.2.1                    pypi_0    pypi
cuda-cccl                 12.4.127                      0    nvidia
cuda-cudart               11.8.89                       0    nvidia
cuda-cudart-dev           11.8.89                       0    nvidia
cuda-cupti                11.8.87                       0    nvidia
cuda-libraries            11.8.0                        0    nvidia
cuda-libraries-dev        11.8.0                        0    nvidia
cuda-nvrtc                11.8.89                       0    nvidia
cuda-nvrtc-dev            11.8.89                       0    nvidia
cuda-nvtx                 11.8.86                       0    nvidia
cuda-profiler-api         12.4.127                      0    nvidia
cuda-runtime              11.8.0                        0    nvidia
cycler                    0.12.1                   pypi_0    pypi
cython                    3.0.10                   pypi_0    pypi
dash                      2.16.1                   pypi_0    pypi
dash-core-components      2.0.0                    pypi_0    pypi
dash-html-components      2.0.0                    pypi_0    pypi
dash-table                5.0.0                    pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
diff-gaussian-rasterization 0.0.0                    pypi_0    pypi
docker-pycreds            0.4.0                    pypi_0    pypi
evo                       1.11.0                   pypi_0    pypi
exceptiongroup            1.2.1                    pypi_0    pypi
executing                 2.0.1                    pypi_0    pypi
fastjsonschema            2.19.1                   pypi_0    pypi
filelock                  3.13.1           py39haa95532_0
flask                     3.0.3                    pypi_0    pypi
fonttools                 4.51.0                   pypi_0    pypi
freetype                  2.12.1               ha860e81_0
gitdb                     4.0.11                   pypi_0    pypi
gitpython                 3.1.43                   pypi_0    pypi
glfw                      2.7.0                    pypi_0    pypi
gmpy2                     2.1.2            py39h7f96b67_0
idna                      3.4              py39haa95532_0
imgviz                    1.7.5                    pypi_0    pypi
importlib-metadata        7.1.0                    pypi_0    pypi
importlib-resources       6.4.0                    pypi_0    pypi
intel-openmp              2023.1.0         h59b6b97_46320
ipython                   8.18.1                   pypi_0    pypi
ipywidgets                8.1.2                    pypi_0    pypi
itsdangerous              2.2.0                    pypi_0    pypi
jedi                      0.19.1                   pypi_0    pypi
jinja2                    3.1.3            py39haa95532_0
jpeg                      9e                   h2bbff1b_1
jsonschema                4.21.1                   pypi_0    pypi
jsonschema-specifications 2023.12.1                pypi_0    pypi
jupyter-core              5.7.2                    pypi_0    pypi
jupyterlab-widgets        3.0.10                   pypi_0    pypi
kiwisolver                1.4.5                    pypi_0    pypi
lerc                      3.0                  hd77b12b_0
libcublas                 11.11.3.6                     0    nvidia
libcublas-dev             11.11.3.6                     0    nvidia
libcufft                  10.9.0.58                     0    nvidia
libcufft-dev              10.9.0.58                     0    nvidia
libcurand                 10.3.5.147                    0    nvidia
libcurand-dev             10.3.5.147                    0    nvidia
libcusolver               11.4.1.48                     0    nvidia
libcusolver-dev           11.4.1.48                     0    nvidia
libcusparse               11.7.5.86                     0    nvidia
libcusparse-dev           11.7.5.86                     0    nvidia
libdeflate                1.17                 h2bbff1b_1
libnpp                    11.8.0.86                     0    nvidia
libnpp-dev                11.8.0.86                     0    nvidia
libnvjpeg                 11.9.0.86                     0    nvidia
libnvjpeg-dev             11.9.0.86                     0    nvidia
libpng                    1.6.39               h8cc25b3_0
libtiff                   4.5.1                hd77b12b_0
libuv                     1.44.2               h2bbff1b_0
libwebp-base              1.3.2                h2bbff1b_0
lightning-utilities       0.11.2                   pypi_0    pypi
lpips                     0.1.4                    pypi_0    pypi
lz4-c                     1.9.4                h2bbff1b_0
markdown-it-py            3.0.0                    pypi_0    pypi
markupsafe                2.1.3            py39h2bbff1b_0
matplotlib                3.8.4                    pypi_0    pypi
matplotlib-inline         0.1.7                    pypi_0    pypi
mdurl                     0.1.2                    pypi_0    pypi
mkl                       2023.1.0         h6b88ed4_46358
mkl-service               2.4.0            py39h2bbff1b_1
mkl_fft                   1.3.8            py39h2bbff1b_0
mkl_random                1.2.4            py39h59b6b97_0
mpc                       1.1.0                h7edee0f_1
mpfr                      4.0.2                h62dcd97_1
mpir                      3.0.0                hec2e145_1
mpmath                    1.3.0            py39haa95532_0
munch                     4.0.0                    pypi_0    pypi
natsort                   8.4.0                    pypi_0    pypi
nbformat                  5.7.0                    pypi_0    pypi
nest-asyncio              1.6.0                    pypi_0    pypi
networkx                  3.1              py39haa95532_0
ninja                     1.12.0               h91493d7_0    conda-forge
numpy                     1.26.4           py39h055cbcc_0
numpy-base                1.26.4           py39h65a83cf_0
open3d                    0.17.0                   pypi_0    pypi
opencv-python             4.8.1.78                 pypi_0    pypi
openjpeg                  2.4.0                h4fc8c34_0
openssl                   3.2.1                hcfcfb64_1    conda-forge
packaging                 24.0                     pypi_0    pypi
pandas                    2.2.2                    pypi_0    pypi
parso                     0.8.4                    pypi_0    pypi
pillow                    10.2.0           py39h2bbff1b_0
pip                       23.3.1           py39haa95532_0
platformdirs              4.2.1                    pypi_0    pypi
plotly                    5.21.0                   pypi_0    pypi
plyfile                   0.8.1              pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.43                   pypi_0    pypi
protobuf                  4.25.3                   pypi_0    pypi
psutil                    5.9.8                    pypi_0    pypi
pure-eval                 0.2.2                    pypi_0    pypi
pyglm                     2.7.1                    pypi_0    pypi
pygments                  2.17.2                   pypi_0    pypi
pyopengl                  3.1.5                    pypi_0    pypi
pyopengl-accelerate       3.1.5                    pypi_0    pypi
pyparsing                 3.1.2                    pypi_0    pypi
pyrealsense2              2.55.1.6486              pypi_0    pypi
pysocks                   1.7.1            py39haa95532_0
python                    3.9.19               h1aa4202_0
python-dateutil           2.9.0.post0              pypi_0    pypi
pytorch                   2.0.1           py3.9_cuda11.8_cudnn8_0    pytorch
pytorch-cuda              11.8                 h24eeafa_5    pytorch
pytorch-mutex             1.0                        cuda    pytorch
pytz                      2024.1                   pypi_0    pypi
pywin32                   306                      pypi_0    pypi
pyyaml                    6.0.1                    pypi_0    pypi
pyzed                     4.1                      pypi_0    pypi
referencing               0.35.0                   pypi_0    pypi
requests                  2.31.0           py39haa95532_1
retrying                  1.3.4                    pypi_0    pypi
rich                      13.7.1                   pypi_0    pypi
rpds-py                   0.18.0                   pypi_0    pypi
ruff                      0.4.2                    pypi_0    pypi
scipy                     1.13.0                   pypi_0    pypi
seaborn                   0.13.2                   pypi_0    pypi
sentry-sdk                2.0.1                    pypi_0    pypi
setproctitle              1.3.3                    pypi_0    pypi
setuptools                68.2.2           py39haa95532_0
simple-knn                0.0.0                    pypi_0    pypi
six                       1.16.0                   pypi_0    pypi
smmap                     5.0.1                    pypi_0    pypi
sqlite                    3.41.2               h2bbff1b_0
stack-data                0.6.3                    pypi_0    pypi
sympy                     1.12             py39haa95532_0
tbb                       2021.8.0             h59b6b97_0
tenacity                  8.2.3                    pypi_0    pypi
torchaudio                2.0.2                    pypi_0    pypi
torchmetrics              1.3.2                    pypi_0    pypi
torchvision               0.15.2                   pypi_0    pypi
tqdm                      4.66.2             pyhd8ed1ab_0    conda-forge
traitlets                 5.14.3                   pypi_0    pypi
trimesh                   4.3.1                    pypi_0    pypi
typing_extensions         4.9.0            py39haa95532_1
tzdata                    2024.1                   pypi_0    pypi
ucrt                      10.0.22621.0         h57928b3_0    conda-forge
urllib3                   2.1.0            py39haa95532_1
vc                        14.2                 h21ff451_1
vc14_runtime              14.38.33130         h82b7239_18    conda-forge
vs2015_runtime            14.38.33130         hcb4865c_18    conda-forge
vs2022_win-64             19.38.33130         h0bfb142_18    conda-forge
vswhere                   3.1.4                h57928b3_0    conda-forge
wandb                     0.16.6                   pypi_0    pypi
wcwidth                   0.2.13                   pypi_0    pypi
werkzeug                  3.0.2                    pypi_0    pypi
wheel                     0.41.2           py39haa95532_0
widgetsnbextension        4.0.10                   pypi_0    pypi
win_inet_pton             1.1.0            py39haa95532_0
xz                        5.4.6                h8cc25b3_0
zipp                      3.18.1                   pypi_0    pypi
zlib                      1.2.13               h8cc25b3_0
zstd                      1.5.5                hd43e919_0

[muskie82]

Hi,

Thank you for your interest in our work.
I think this is the same issue as #20 #80 , which comes from the bug of PyTorch multiprocess module for Windows.
I don't find the solution yet, thus I recommend to build a Ubuntu virtual environment on docker.

Best,

[TonyDua]

Hi,

Thank you for your interest in our work. I think this is the same issue as #20 #80 , which comes from the bug of PyTorch multiprocess module for Windows. I don't find the solution yet, thus I recommend to build a Ubuntu virtual environment on docker.

Best,

Thank you for your reply, I will try to deploy and test on Ubuntu. I will come back here later to update my progress.