ByteTrack memory consumption increases indefinitely

[tc360950]

Search before asking

• I have searched the Supervision issues and found no similar bug report.

Bug

ByteTrack stores removed tracklets in a list, which is extended with new removed tracklets after every pass (line 489 in byte_tracker/core.py:

self.lost_tracks = sub_tracks(self.lost_tracks, self.removed_tracks)
self.removed_tracks.extend(removed_stracks)

As a result, size of removed_tracks collection grows unbounded and computational cost of
sub_tracks(self.lost_tracks, self.removed_tracks) increases with every pass.
The fix is simple - we only need to keep latest (i.e. the one from the last pass) removed tracklets in the collection - I've submitted a corresponding PR.

Environment

Not applicable

Minimal Reproducible Example

This simple script shows that update efficiency decreases steadily:

import random
import time
from typing import Generator

import matplotlib.pyplot as plt
import numpy as np

from supervision import ByteTrack, Detections

byte_track = ByteTrack(
    track_activation_threshold=0.25,
    lost_track_buffer=30,
    minimum_matching_threshold=0.8,
    frame_rate=30,
    minimum_consecutive_frames=2,
)


def get_detections() -> Generator[Detections, None, None]:
    start_bboxes = [
        [100, 100, 150, 200],
        [200, 100, 250, 200],
        [400, 100, 450, 200],
        [500, 100, 550, 200],
        [600, 100, 650, 200],
    ]
    bbox_count = len(start_bboxes)
    while True:
        bboxes = np.array(start_bboxes.copy()).reshape((-1, 4))
        for i in range(0, 10):
            for j in range(0, bbox_count):
                bboxes[j][0] -= i * 10
                bboxes[j][2] -= i * 10
            yield Detections(
                xyxy=bboxes,
                confidence=np.array([random.random() for _ in range(0, bbox_count)]),
                class_id=np.array([0 for _ in range(0, bbox_count)]),
            )


update_times = []
removed_tracks_count = []
detections_generator = get_detections()
for _ in range(0, 50000):
    detections = next(detections_generator)
    start = time.time()
    byte_track.update_with_detections(detections)
    end = time.time()
    removed_tracks_count.append(len(byte_track.removed_tracks))
    update_times.append(end - start)


plt.plot(range(0, len(update_times)), update_times)
plt.savefig("update_times.png")
plt.plot(range(0, len(removed_tracks_count)), removed_tracks_count)
plt.savefig("removed_tracks.png")

Additional

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[SkalskiP]

Hi, @tc360950! 👋🏻 I completely agree that the problem exists. Why do you think storing values from the last pass is enough?

[tc360950]

Hi @SkalskiP, removed_tracks collection is only used on line 489

self.lost_tracks = sub_tracks(self.lost_tracks, self.removed_tracks)

where it's used to get rid of lost tracks which have been marked for removal (i.e. those which have not been matched to a detection for a predefined time). Since tracks marked for the removal in the latest pass are added to removed_tracks collection on line 490

self.removed_tracks.extend(removed_stracks)

lost tracks which were removed on pass no. N will actually be removed (i.e. moved to removed_tracks collection) after pass no. N + 1 (which btw is another error, although without any dire consequences). This implies that we must at least do (which is what I propose):

self.removed_tracks = removed_stracks

Now, as to why this is enough. Once a track lands in removed_tracks it does not get out anywhere. I.e. once track is removed from lost_tracks on line 489, it will not reappear there. As a result we do not need to keep whole history, only most recently removed tracks are needed to "sanitize" lost_tracks collection.

I agree that it's a risky change, since there are no unit tests for ByteTrack (I can of course come up with some scripts which compare results from both versions on hand crafted or random data (which I did before raising this issue) but it still seems "not quite right"). There is an open PR with unit tests for ByteTrack. I have not looked at it but maybe that's the best way to introduce this change - add good unit tests for ByteTrack.

[SkalskiP]

Hi @tc360950 👋🏻 Thank you for explaining your thought process. The ByteTrack code was transferred to Supervision (with minor changes) from another codebase, which is why it slightly deviates from the writing style we have in Supervision and, as you rightly observed, is not tested at all. Thanks a lot for the analysis you conducted.

[SkalskiP]

I'm closing this PR as #1166 was just merged.