[RichLabelAnnotator] - add support for unicode labels

[Ying-Kang]

Description

LabelAnnotator uses OpenCV as a rendering engine. Unfortunately, cv2.putText, which we use underneath, only supports ASCII characters. A solution to this problem would be the implementation of a new annotator that adds support for Unicode characters based on Pillow.

We previously considered a similar idea, but we decided not to implement it due to time constraints. #159

API

class RichLabelAnnotator:

    def __init__(
        self,
        color: Union[Color, ColorPalette] = ColorPalette.DEFAULT,
        text_color: Color = Color.WHITE,
        font_path: str = "path/to/default/font.ttf",
        font_size: int = 10,
        text_padding: int = 10,
        text_position: Position = Position.TOP_LEFT,
        color_lookup: ColorLookup = ColorLookup.CLASS,
    ):
        pass

    def annotate(
        self,
        scene: ImageType,
        detections: Detections,
        labels: List[str] = None,
        custom_color_lookup: Optional[np.ndarray] = None,
    ) -> ImageType:
       pass

Additional

• Note: Please share a Google Colab with minimal code to test the new feature. We know it's additional work, but it will speed up the review process. The reviewer must test each change. Setting up a local environment to do this is time-consuming. Please ensure that Google Colab can be accessed without any issues (make it public). Thank you! 🙏🏻

[SkalskiP]

Hi @Ying-Kang 👋🏻, thanks for your interest in supervision.

We initially wanted to add support for Unicode labels, but we ultimately became overwhelmed by the work involved in working around other annotators and decided to limit the scope.

I think it would be a good idea to revisit this idea now. We already have the framework ready we would just have to add this new functionality.

[SkalskiP]

Hi @Ying-Kang 👋🏻 I'll keep it open if that's okey with you.

[jeslinpjames]

Hi 👋, I'd like to try working on this.

[SkalskiP]

Hi @jeslinpjames 👋🏻 ! I assigned the issue to you. Good luck!

[onuralpszr]

@SkalskiP @jeslinpjames

I did research previously, to use "Unicode" support to visualize Opencv we need to make sure it needs to be compiled with extra parameters (PIP version of OpenCV-python doesn't enable it because of licensing) but in the next version of OpenCV (5.0) will change and enable by default.

They will introduce TrueType font support (opencv/opencv#18760) and I noticed they are using "Rubik" font and support many languages as well.

For Pillow, we need to use an external font file based on OpenCV's choice maybe we can use that font until they release version 5.0. That's all I saw and see for now. I hope that helps bit more in case to find something as well.

[jeslinpjames]

Hi @SkalskiP,

🙌 Thanks for the assignment!

And @onuralpszr, appreciate the info on Unicode support.

[jeslinpjames]

Hi @onuralpszr,

Initially, I attempted to use the Rubik-regular.ttf font file, but it didn't effectively support all languages. Then, I found that the Arial Unicode Font.ttf file provides better support for a wide range of languages.

I also had a question regarding the default value for the font_path parameter. In the __init__ method of the RichLabelAnnotator class, I've set the default value of font_path to None. This allows users to either specify a custom font path or rely on the system font if no path is provided.

Here's the updated snippet of the __init__ method:

class RichLabelAnnotator:
    def __init__(
        self,
        color: Union[Color, ColorPalette] = ColorPalette.DEFAULT,
        text_color: Color = Color.WHITE,
        font_path: str = None,
        font_size: int = 10,
        text_padding: int = 10,
        text_position: Position = Position.TOP_LEFT,
        color_lookup: ColorLookup = ColorLookup.CLASS,
        border_radius: int = 0,
    ):
        self.color = color
        self.text_color = text_color
        self.text_padding = text_padding
        self.text_anchor = text_position
        self.color_lookup = color_lookup
        self.border_radius = border_radius
        
        if font_path is not None:
            try:
                self.font = ImageFont.truetype(font_path, font_size)
            except OSError:
                print(f"Font path '{font_path}' not found. Using a system font.")
                self.font = ImageFont.load_default(size=font_size)
        else:
            self.font = ImageFont.load_default(size=font_size)

Please let me know if any further adjustments are needed.

[onuralpszr]

Hello @jeslinpjames

What do you mean by using "system font"? Are you going to use font from OS (Linux, Windows, macOS) or use font from Pillow, OpenCV font?

I am debating whether should we even need RichLabelAnnotator maybe we can consider making the change to labelAnnotator and If you provide a font path it will just use that font and no try except require plus less code work.

Another part I am thinking of is maybe we can also create another subpackage for supervision addon to provide fonts in Python package by installing via pip and without manually typing the full path of the font (but the license needs to be checked)

@SkalskiP @LinasKo what do you guys think?

Should we extend our LabelAnnotator or add a completely new annotator?
Should we create font packages like assets we have in supervision for providing easy font support (even if we can't add font directly, we can add URL and download)

[jeslinpjames]

Hello @jeslinpjames

What do you mean by using "system font"? Are you going to use font from OS (Linux, Windows, macOS) or use font from Pillow, OpenCV font?

I am debating whether should we even need RichLabelAnnotator maybe we can consider making the change to labelAnnotator and If you provide a font path it will just use that font and no try except require plus less code work.

Another part I am thinking of is maybe we can also create another subpackage for supervision addon to provide fonts in Python package by installing via pip and without manually typing the full path of the font (but the license needs to be checked)

@SkalskiP @LinasKo what do you guys think?

Should we extend our LabelAnnotator or add a completely new annotator? Should we create font packages like assets we have in supervision for providing easy font support (even if we can't add font directly, we can add URL and download)

The font provided by Pillow, and I've almost completed the RichLabelAnnotator. Let me know how you'd like me to change it up.

[LinasKo]

I like Onuralp's idea about merging this with the standard label annotator, but I don't know whether that would break anything in systems currently in use.

If we can apply these changes to LabelAnnotator without such issues, I think that'd be perfect.
But let's see how RichLabelAnnotator looks, review that, and then decide.

[jeslinpjames]

I like Onural's idea about merging this with the standard label annotator, but I don't know whether that would break anything in systems currently in use.

If we can apply these changes to LabelAnnotator without such issues, I think that'd be perfect. But let's see how RichLabelAnnotator looks, review that, and then decide.

Here is the Google Colab link: Google Colab Notebook

[LinasKo]

I'll see if I can review in the next few days.
(Though we're pretty short on time lately)

[onuralpszr]

I'll see if I can review in the next few days.
(Though we're pretty short on time lately)

If you may I can handle it

[onuralpszr]

I'll see if I can review in the next few days.
(Though we're pretty short on time lately)

If you may I can handle it

I just need opinion about merge into label without api break or separate annotator that's all

[LinasKo]

If you may I can handle it

That would be most helpful :)

I just need opinion about merge into label without api break or separate annotator that's all

I'll try to find the time tomorrow.

[onuralpszr]

If you may I can handle it

That would be most helpful :)

I just need opinion about merge into label without api break or separate annotator that's all

I'll try to find the time tomorrow.

My believe we can handle without api break and just one extra if statement needed if font stated without font it is gonna use what we have so, we can do minimal change as well

[onuralpszr]

@jeslinpjames please open PR for RichLabelAnnotator as a new Annotator, take out "resolve_text_background_xyxy" and put utils.py both LabelAnnotators should be able to use it. Let's continue to rest in PR