[Bug report] Android : Object tracking and video playback

[lmartin]

I'm trying to use object tracking for video playback use case, but after some tests it looks like that object tracking is only designed for live video, correct?

It looks like there is a kind of latency threshold between 2 tracking frames, beyond this threshold tracker failed to find any objects.

I found this project https://github.com/ZeroOneZeroR/android-ml-kit-vision-with-video but is does not use object tracking with video playback.

Is there a way to configure the tracker to accept any latency between 2 frames?

[ai-plays]

Your observation is right. There is some internal threshold that if two frames are too far apart in time, we don't try to apply the tracker algorithm between them (because things may change more dramatically that the tracker will not work well).

Are you using live video feed or processing an already recored video? Could you tell us more why you need to increase the threshold? Because you are running on some very low end devices that is very slow?

[lmartin]

I'm using video playback of an already recorded video. My app is a video editor. I want to track objects on frames each second.

To do this : I pause the video, process the frame, seek 1 second later, and so on... but video seeking can be long (depending on device and video input) and furthermore I use Bitmap API in my case (which is even longer).

Having the possibility to set the threshold value would be great.

[ai-plays]

If the frame interval is 1 second, the position of the same object in the two frames could be very different. In such case the tracker algorithm may not work well even if you adjust the threshold.

If you really want to do it, there is a workaround you could try: preprocess the frames in advance to get the sequence of frames you want to run object detection and tracking on, then just pass them one by one to the detector without seeking in between.

[lmartin]

Another workaround that would fit the tracker algorithm would be to preprocess all the frames in 1 second interval and then pass them to the detector, I could repeat this sequence on all the video length but it will need to disable the threshold to prevent failing between 2 sequences.

[lmartin]

Would it possible to expose the threshold as an editable property?

[ai-plays]

Hi lmartin,

There is no release plan for this feature in the near future, for this specific request.

I would suggest to go with some workaround with what is available at the moment.

Sorry for the inconvenience.