[discussion / planning] How to move forward with full ONNX support

[felixdittrich92]

Hi @frgfm @charlesmindee and all other 👋,

we will be soon in an state with the library where we are able to export the models into onnx format. (full support planned 0.6)
So i think it would be good to discuss some further work in this topic.

As a user i want:

• export models into onnx to use it into a production system
• load the exported models directly in the library like ocr_predictor without installing PyTorch or Tensorflow as dependency onnxruntime should be enough
• be able to configure the runntime (for example CPUExecutionProvider / GPUExecutionProvider)

There are some open question:

• How want we to design the loading ? I think we have to encapsulate it from the default ocr_predictor maybe introducing a onnx_ocr_predictor makes sense ?
• We need to do the post processing afterwards so any ideas how we can adopt this ?
• We need to store some information about the onnx exported models for example which architecture, vocab, and so on .. maybe working with config files in the same way we have done this for huggingface push_to_hub and from_hub ?

Feel free to add some additional to this list if i missed anything 🤗

Any feedback is very very welcome ❤️

[charlesmindee]

Hi Felix,
Thanks for bringing this on the table, I like the idea of having a onnx_ocr_predictor or something equivalent that won't require tensorflow nor torch. For the postprocessing, it would be great to share it with the tensorflow/torch models since it shouldn't be DL-framework related, WDYT ?

[frgfm]

Alright, but docTR, as is, is not light. Meaning that it's not exactly optimal for production.
Even without the DL backend like Pytorch or TF, docTR comes with a lot of dependencies that might not be relevant for production situations 🤷‍♂️

But either way, this is a very large topic: balance of docTR between dev / user types, dev / production situations

[felixdittrich92]

Yes i agree, i would say in a first step it would be good to start with a PoC after the 0.5.2 release and the TF mobilenet arch fix.
If this is done we can further check if it makes sense/ is possible to provide maybe a minimal build. wdyt ?

(And yes, it would be easiest if we managed to make the postprocessing onnx compatible, then everything else would be much easier and more flexible. But as you said, there are quite a few places that can break our necks 🤕 😅)

On the other hand in this case docTR would be the first library which provides End-to-End OCR in an full Onnxruntime environment (or do you know anyone who can provide something ?) which would definitely be a blatant unique point

[frgfm]

We got a few options, the easiest one for us would be to have a docTR core inference build: ONNX centered, no dev features, only inference (I.e. for production)

So for instance, the post processing can be left to the user to perform and we only provide the logits (not optimal I agree)
Going for full ONNX compatibility is going to be very long

My point is that:

• docTR can natively provide ONNX export
• but providing ONNX inference feature without discarding dev features will make any production docker between 3 and 5 times larger than what it needs (mostly because of Pytorch & TensorFlow)
• so I suggest considering the ONNX version something to implement for a production situation and the best way I can see for this: implement it in the API boilerplate that we provide. I've already done this for personal projects, it's blazing fast, so much lighter. The only trick is that I don't have postprocessing that is this complex.

[felixdittrich92]

@frgfm yes but in this case we ignore the biggest advantage to get an well structured output which is easy to use . I think if we find a solution for the pre- and post-processing it is also possible to have a lightweight build only for inference also inside docTR. I will test something after we are done with 0.5.2 and i'm back from vacation 😊

[frgfm]

Yup, ideally, if we could redesign everything instantly:

• most independent modules should be set as extras
• DL backend: for dev & training, PyTorch and TF should be available. for inference, also have ONNX
• post-processing: everything should be implemented in numpy, PyTorch or TF (for the last two, that can be passed to ONNX then)

But we need to find a middleground here :)

[ajkdrag]

@felixdittrich92 I see the docs is updated to export to Onnx, but is the inference/loading working as of now?

[felixT2K]

@ajkdrag pre-processing should not be the problem. But post-processing is tricky because especially for the detection we use some shapely and opencv code

[nospotfer]

Do we really need to convert the post-processing pipeline to ONNX as well? As I user, I'd be more than happy to have a predictor or even different .onnx models (detection.onnx, recognition.onnx, etc) and leave the post-processing as methods depending on shapely, opencv and any other libraries.

[felixdittrich92]

@nospotfer Good time to ask :D

See: https://github.com/felixdittrich92/OnnxTR (I have some problems with the gpu installation atm but the cpu version works well already)

[felixdittrich92]

@odulcy-mindee Maybe we can discuss later on if we want to move the repo to the mindee org otherwise keeping it on my private account would also be fine - it's much better to keep this seperated instead of having a third backend in doctr i think
(I will take care to keep it in sync with doctr)

[nospotfer]

Wow! Perfect timing!
I'll have a look. GPU inference is a nice to have, but I understand most of the people (including myself) are more than happy with a CPU inference pipeline.

[felixdittrich92]

I think we can close the discussion now with https://github.com/felixdittrich92/OnnxTR :)