[FR] load annotations with specific status

[mmoollllee]

Proposal Summary

Adding an annotations job with segement_size to CVAT, makes me want to load those segments with e.g. CVAT Job Stage "validation" back into FiftyOne to validate the progress.

Motivation

• As I add annotation jobs with predictions to CVAT, where those predictions become ground_truth annotations, I don't want to load the partial progress into the new dest_field. It helps me to validate progress in FiftyOne instead of CVAT, and maybe use this progress for training already.
• There's currently no option to filter load_annotations

What areas of FiftyOne does this feature affect?

• App: FiftyOne application
• Core: Core fiftyone Python library
• Server: FiftyOne server

Details

I'm adding annotation jobs to CVAT like this:

view.annotate(
    anno_key,
    label_field="bad_predictions",
    segment_size=50
)

and I'd like to have a filter_jobs key for load_annotations, e.g.:

dataset.load_annotations(
    anno_key, 
    dest_field="ground_truth", 
    unexpected="keep",
    filter_jobs="validation" # only import CVAT jobs with "stage: validation"
)

Willingness to contribute

The FiftyOne Community welcomes contributions! Would you or another member of your organization be willing to contribute an implementation of this feature?

• Yes. I can contribute this feature independently
• Yes. I would be willing to contribute this feature with guidance from the FiftyOne community
• No. I cannot contribute this feature at this time

[brimoor]

@mmoollllee this feature request makes sense to me 👍

In order to implement this, you'd need to update the download_annotations() interface to allow for the possibility of passing backend-specific **kwargs to customize how the annotations are downloaded.

Then, since the load_annotations() method is generic (supports multiple backends), you'd need to check its **kwargs and pull out any download parameters for the specific backend being used.

We have an extract_kwargs_for_function() utility that can do that 👍

Let me know if this makes sense and happy to provide additional guidance if I can be of assistance 🤗

[mmoollllee]

Happy to hear the feature makes sense :))

Puhh I digged around the provided code parts and need to accept that I'm totally lost in this advanced coding architecture... I guess this is beyond my abilities :(
Maybe I'll give it another try and a lot of testing...
Otherwise it seems like somebody else needs to implement this. Sorry :(

[brimoor]

I believe in you! If you're able to add your parameter only in this location, then we can help complete the feature for you 🤗

fiftyone/fiftyone/utils/cvat.py

Line 4562 in 7853d08

def download_annotations(self, results):

[mmoollllee]

Thanks for your pushing me!! :)

See my current status in this commit

• Get argument job_status. No progress with this, but I tried this:

fou.extract_kwargs_for_function(foua.load_annotations, self.kwargs)

but when I add a argument to .load_annotations I get TypeError: load_credentials() got an unexpected keyword argument 'job_status'. Don't know what to do :(

• If job_status is one of CVAT job_status build a list of frames to include:

                # If job_status is set and is one of CVAT's status'
                frames_to_download = []
                if job_status in ["annotation", "validation", "acceptance"]:
                    for job_id in job_ids[task_id]:
                        # Get job_status
                        job_url = self.taskless_job_url(job_id)
                        try:
                            job_resp = self.get(job_url).json()
                        except:
                            logger.warning(
                                "Couldn't fetch job information. Skipping"
                            )
                            continue

                        if not job_resp["status"] == job_status:
                            logger.info(
                                "Skipping job %s because of status mismatch",
                                job_id,
                            )
                            continue

                        frames_to_download += range(job_resp["start_frame"], job_resp["stop_frame"])

• After fetching data_resp filter shapes, tags & tracks to match only frames_to_download:

                if frames_to_download:
                    all_shapes = [x for x in all_shapes if x['frame'] in frames_to_download]
                    all_tags = [x for x in all_tags if x['frame'] in frames_to_download]
                    all_tracks = [x for x in all_tracks if x['frame'] in frames_to_download]

I have to say I'm not really confident in what I did. For example I never worked with videos in CVAT. Will this still work for videos?

I'd be happy to get feedback and another hint on how to pass job_status from dataset.load_annotations!