Object Detection Overlay on Both Camera Images

[keli95566]

Preliminary Checks

• This issue is not a duplicate. Before opening a new issue, please search existing issues.
• This issue is not a question, bug report, or anything other than a feature request directly related to this project.

Proposal

Having the object detection or skeleton overlay on both camera image streams. The current version only supports overlay on a single image.

Use-Case

No response

Anything else?

No response

[keli95566]

@Myzhar Hi there! Is there a fast way that I could get this to work on my own?

[obraun-sl]

Hi @keli95566

You can recontruct the 2D points from skeleton or object detection by calculating the 2D points in the right image from the 2D point in the left image + the disparity (can be extracted from the depth map).

If you have a 2D point in the left image (uL,vL), and the 3D coordinate of this point (X,Y,Z)

the 2D point in the right image (uR,vR) will be :

vR = vL;
uR = uL - (focale * baseline) / Z ; 

focale can be extracted from zed.getCameraInformation().calibration_parameters.left_cam.fx
baseline can be extracted from zed.getCameraInformation().calibration_parameters.getCameraBaseline();

( basically disparity = focale x Baseline / Depth. )

[keli95566]

Hi there! Just some update, the following pipeline using zeddemux works for adding overlay to both eyes without having to add new scripts. One additional small question: now the maximum framerate is only 15fps even for only HD720 resolution, is it possible to scale up the resolution to 30 fps?

gst-launch-1.0 zedsrc stream-type=2 od-enabled=true od-detection-model=0 camera-resolution=2 camera-fps=30 ! queue ! zeddemux is-depth=false name=demux demux.src_left! queue ! zedodoverlay ! autovideoconvert ! omxh264enc! 'video/x-h264, stream-format=(string)byte-stream' ! h264parse ! rtph264pay config-interval=-1 pt=96 ! queue ! udpsink clients=[IP] max-bitrate=30000000 sync=false async=false demux.src_aux ! queue ! zedodoverlay ! autovideoconvert ! omxh264enc ! 'video/x-h264, stream-format=(string)byte-stream' ! h264parse ! rtph264pay config-interval=-1 pt=96 ! queue ! udpsink clients=[IP] max-bitrate=30000000 sync=false async=false

[Myzhar]

The maximum framerate for 720p is 60 Hz.
If you cannot get more than 15 FPS it means that you r system cannot handle it, or the network bandwidth is not enough.

[keli95566]

I am wondering what is the object detection based on? Is it YOLO? is there any way we could customize the model/label based on gstreamer? Thank you very much!

[Myzhar]

The Object Detection module uses internal models optimized to take advantage of the depth information.
The models cannot be retrained to add support for custom objects, but the ZED SDK can track objects detected with custom detectors:
https://www.stereolabs.com/docs/object-detection/custom-od/

[keli95566]

I see, thank you! However, I see that there is also a ZED Plugin for Yolo ( both c++ and Python) : https://www.stereolabs.com/docs/yolo/
Is it worth a try to connect GStreamer to the ZED Yolo plugin? Is real-time inference possible, and has the performance for ZED Yolo been benchmarked for the Jetson TX2 Zed box?