New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feature(frameworks): bentoml.onnx runner accept kwargs #3561
base: main
Are you sure you want to change the base?
Conversation
Codecov Report
@@ Coverage Diff @@
## main #3561 +/- ##
=====================================
Coverage 0.00% 0.00%
=====================================
Files 154 154
Lines 12620 12632 +12
=====================================
- Misses 12620 12632 +12
|
cc @larme, I ran the test locally, it passed, but it still didn't run on CI. can you also fix it as well? |
return _process_output(raw_outs) | ||
if options.use_kwargs_inputs: | ||
|
||
def _run(self: ONNXRunnable, **kwargs: ONNXArgType) -> t.Any: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this mean we can not call onnx_runner.run(arg1, arg2, kwarg1=xxx)
, using args and kwargs at same time?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no I don't think so.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it does.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For every onnx model we have a fixed list of inputs with their own names. Calling by args means refer each argument by its position while calling by kwargs means refer each argument by its name. Now if we mix these 2 methods, I feel that it's very easy for user to miss some arguments.
Calling by kwargs is useful when dealing with NLP models where the tokenizer output a dictionary and the dictionary is passed to the model as kwargs. In this use case there's no additional positional argument to be passed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need more discussion about this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
let's have a discussion on engineer meeting
b565287
to
591dcd5
Compare
@aarnphm Now the tests run on CI. The problem is that the new bert tests will missing the pytorch dependencies (I thought installing transformers will install torch by default, I was wrong). |
a468d5b
to
a90f59f
Compare
a90f59f
to
607ec46
Compare
This has been on this list for a while now @larme we should circle back and get this one merge |
What does this PR address?
Sometimes it's more nature to call onnx model with keyword arguments. For example, bert's tokenizer will output a dictionary, which the original model will call with
model(**input)
. Our runner should simulate this behaviorFixes #(issue)
Before submitting:
guide on how to create a pull request.
make format
andmake lint
script have passed (instructions)?those accordingly? Here are documentation guidelines and tips on writting docs.