Polygon tracking feature with XMem tracker

[omerferhatt]

Motivation and context

Polygon tracking is a crucial part of computer vision annotation tasks. It helps the required consecutive frame annotation, shortens the required annotation time, creates more precise annotation and moreover simplifies all HITL (Human in the loop) processes. Since the CVAT doesn't support this feature, it seems that adding this functionality with side benefits will be good for the sake of this project.

It aims to solve the below problems:

• Using polygon tracking algorithms natively inside the CVAT to use it with semi-automatic annotations.
• Tracking multiple polygon objects without losing the common track options (enable/disable, modify the last tracked object and continue from it, etc.)
• Creating trackable polygon objects through interactor's (which allows to use SAM with any kind of polygon tracking algorithm)

According to the previous issues, it:

• Fixes Enhancement: XMem mask video propagation as CVAT AI Tools polygon tracker #5465
• Fixes Feature Idea: Incorporate "Segment Anything" #5984 (comment)
• Fixes Feature Idea: Incorporate "Segment Anything" #5984 (comment)
• Helps Support for Multi-Object Multi-Camera Tracking #4915
• Helps Allow tracker to run on more frames automatically #5686

How it looks

initial_demo_video.webm

• SAM runs on the CPU and XMem tracker runs on the GPU

Resource usage

• For 5 object tracking on 1280x720 video will use approx. 2GB VRAM at peak.

Known issues and further optimization cases:

• When maxWorker is more than 1 in function.yaml, the entire tracker works again in a duplicate manner. Since the problem is not solved, it is set to 1 for now.
• Since tracking works on individual objects, it can be improved with batch inference.
• As explained in step 2, this increases the request size since model states are kept separately for each input. Therefore, the process outside the model slows down as the number of objects tracked increases.

How has this been tested?

Tested and developed according to CVAT documentation

Checklist

• I submit my changes into the develop branch
• I have created a changelog fragment
• I have updated the documentation accordingly
• I have added tests to cover my changes
• I have linked related issues (see GitHub docs)
• I have increased versions of npm packages if it is necessary
  (cvat-canvas,
  cvat-core,
  cvat-data and
  cvat-ui)

License

• I submit my code changes under the same MIT License that covers the project.
  Feel free to contact the maintainers if that's a concern.

Summary by CodeRabbit

• New Features

  • Added object type selection functionality to enhance user control over shape and polygon annotations.
  • Introduced new serverless functions for advanced object segmentation using a PyTorch-based model with GPU support.

• Enhancements

  • Improved filtering logic to support both rectangles and polygons more effectively in the annotation interface.

• Documentation

  • Updated component documentation to reflect new functionalities and logic adjustments.

• Performance

  • Optimized image processing and object tracking within the serverless PyTorch model environment.

[coderabbitai]

Walkthrough

The recent updates involve integrating advanced object segmentation capabilities into the CVAT tool, enhancing its ability to handle complex video annotation tasks. This includes the addition of the XMem model for long-term video object segmentation and updates to the UI for better handling of object types during annotation. These changes aim to improve the efficiency and versatility of the annotation process.

Changes

File Path	Change Summary
cvat-ui/.../tools-control.tsx	Updated UI components for object type selection and handling in annotations.
serverless/pytorch/omerferhatt/xmem/nuclio/...	Introduced serverless configurations and handlers for the XMem model, enabling advanced video object segmentation.

Assessment against linked issues

Objective	Addressed	Explanation
Enhancement: XMem mask video propagation as CVAT AI Tools polygon tracker (#5465)	✅
Feature Idea: Incorporate "Segment Anything" (#5984)	❌	The changes do not mention integration or adaptation of the "Segment Anything" model.

Poem

🐰 Oh hop! A leap in tech so grand,
With shapes and masks at our command.
From frame to frame, the objects glide,
Segmented neatly, side by side.
A rabbit's touch in lines of code,
In CVAT, our efforts showed. 🌟
Let's celebrate, for progress made,
In AI's shade, our tools upgrade.

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

• X
• Mastodon
• Reddit
• LinkedIn

Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai generate interesting stats about this repository and render them as a table.
  • @coderabbitai show all the console.log statements in this repository.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

Actionable comments posted: 2

[PMazarovich]

Hello, @bsekachev , do you think this MR will be reviewed/merged soon?
This sounds very helpful for annotators
@omerferhatt thanks!

[bsekachev]

I will try to review it on this week.