-
Notifications
You must be signed in to change notification settings - Fork 404
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG]: Unable to deploy a ML model locally to MLFlow #2235
Comments
The inconsistent type naming between the model deployer registration and the
As it's defined in ServiceType as:
Or we can change the type in src\zenml\integrations\mlflow\services\mlflow_deployment.py on line 128 to just |
@Vishal-Padia thanks for your response.
for your 2nd solution, I tried it but it didn't work. |
@PriyanshBhardwaj |
zenml
zenml
Hello @PriyanshBhardwaj , sorry about the delay! As I learned from the info provided you install Can you do the following and retest? pip3 uninstall mlflow
zenml integration install mlflow -y Moreover, forking on MacOS might not be working always smoothly, if |
System Information
python = 3.9
zenml version = 0.53.1
os = macos
integration = mlflow (downloaded separately by pip install mlflow)
What happened?
Unable to deploy a ml model locally in mlflow. The problem lies in class
MLFlowDeploymentService
in filemlflow_deployment.py
.Please check the reproduction steps to understand the issue clearly.
Reproduction steps
I followed all steps correctly, set the experiment tracker, model deployer and created the stack:
i created a pipeline which will ingest data, train the model, evaluate performance and then deploy model after passing the trigger:
I debugged it, everything is working good and the model is also passing the deployment trigger but the model deployer is not working properly. the problem is with this step:
when pipeline calls it, the log which prints is:
this gets logged from function
deploy_model()
which is in the file/zenml/integrations/mlflow/model_deployers/mlflow_model_deployer.py
. In the same function at line 210 it callsservice.start()
which is in the file/zenml/services/local/local_service.py
and in the same under the start function it logsStarting service 'MLFlowDeploymentService[2ade1153-7fd3-45d1-8ecd-412f86b264b5] (type: model-serving, flavor: mlflow)'.
from line 387 and then when it callsif not self.poll_service_status(timeout):
at line 391 it logs error:when i visited the log file it says:
it raises this issue from here:
/zenml/services/service_registry.py:193 in load_service_from_dict
but in the file
mlflow_model_deployer.py
at line 187 service gets its value from hereservice = cast(MLFlowDeploymentService, existing_service)
which isMLFlowDeploymentService[2ade1153-7fd3-45d1-8ecd-412f86b264b5] (type: model-serving, flavor: mlflow)
.but in class
MLFlowDeploymentService
in filemlflow_deployment.py
at line 128 type is already defined as "model-serving" which i think cant be changed:It is getting timed out due to this because
MLFlowDeploymentService[2ade1153-7fd3-45d1-8ecd-412f86b264b5] (type: model-serving, flavor: mlflow)
is not starting and it will always give timeout error doesn't matter what will be the value of timeout.so why it is giving this error in log file
I tried everything to solve it:
created new stack from scratch
created new pipeline in different stacks
initialize zenml again by deleting .zen folder and again calling
zenml init
in terminal in same directoryi also tried to use
--type=mlflow
while creating model deployer as mentioned somewhere in your old docs in this command:It obviously didnt work.
Nothing worked and also your docs doesnt have a solution for such problems.
My issue is I tried everything to debug this but not able to deploy my model locally to mlflow bcz i cant change type in your internal class which is the root cause. Please resolve the issue and please update your logs and make them more clear for the users.
P.S: model size 1000 bytes only, timeout 60, 120 didnt work for both values
I'm using a separate env for this.
latest version of zenml and mlflow
Relevant log output
Code of Conduct
The text was updated successfully, but these errors were encountered: