Deployment on custom IP, problem with worker connection #391
Comments
|
Hi @MrStrzelec, Have you tried to start with If you would like to run each part of architecture by your own I would recommend checking docker entrypoint script: https://github.com/mljar/mercury/blob/main/docker/mercury/entrypoint.sh In your case, looks like you are missing: Please let me know if it works for you. |
|
Hey! Server working. Okt 30 09:22:45 Servername celery[1155]: return request("get", url, params=params, **kwargs) |
|
I have no idea right now what might be the issue. Is it running on 127.0.0.1? |
|
It's running on a diffrent IP address. I thought i add this information. python manage.py runserver IPADDRESS:PORT |
|
Could you please add |
|
user@server:/opt/jupyterhub/lib/python3.10/site-packages/mercury$ python manage.py runsever --verbose IP:PORT The above exception was the direct cause of the following exception: Traceback (most recent call last): It tried to run also mercury run --verbose IP:PORT and output is similar |
|
Looks like process that is running server doesnt have permission to write logs to the directory. This might be connected to #384 I will need some time to make it configurable. Have you tried to deploy with docker-compose? Have you already developed the notebook with mercury and deployment is the last step? |
pplonski
changed the title
|
I've just changed permissions to this file for everyone and now it's works. Server running correctly and worker working correctly. W/O docker or huggingface. For some reasons Mercury showing that he cannot find this file, while JupyterNotebook working correctly. U know why?
|
|
Great that it is working! Do you have data file in the same directory as notebook? It should read file if it is in the same directory. Alternative might be to provide full path to the file. |
|
Hey! Sorry i had a day off yesterday. Yep, data is in the same directory as a notebook. I also tried with full path and also didn't worked. Any other idea? |
|
This is strange, could you provide full error message? Maybe try to print out the working directory before loading files, just to check the path. Try to add code: import os
print(os.getcwd()) |
|
I think worker was not working at all, i've checked logs and found this: I think worker wouldn't find this file due to having problem with loading notebook. |
|
Do you have notebook file in the same directory in which you start worker? |
|
Hi I have created a private fork without any changes and deployed for http via docker-compose on my local computer with default setting and changed nothing but I am getting similar errors to this #391 (comment) I have used mercury-deploy-demo repo for notebooks and everything seems to be working. admin panel, welcome.md , and notebook icons in welcome page are getting displayed properly but when I click on any of notebooks I get following errors in terminal and I am redirected to notebooks page and notebook is visible in the background but page stays in loading state forever what should I do ? |
|
Hello folks, face the same issue like @keyvan-najafy |
|
Hi @keyvan-najafy, @kskadart, What operating system are you using? How to reproduce your environment? |
|
here is my environment:
An update on the issue:when I change the MERCURY_SERVER_URL in mercury/apps/nbworker/rest.py, and set it to http:\localhost:9000 or http:\mercury:9000 (instead default http://localhost:8000) I get passed the refused connection error but I get another error regarding websocket: so according to this error I get passed mercury/apps/nbworker/rest.py connection error and load notebooks but now web sockets cant connect anymore. |
|
Please try to set |
|
@pplonski adding REACT_APP_SERVER_WS did not help. so basicly the idea is that somehow either by enterypoint.sh or docker-compose.yml there is a mismatching for ports given to the mercury container
This is just a temporary fix as it is pretty bad practice to change I am planning to deploy this on a local cloud platform which does not support docker compose (so I should put all containers separately). I hope these fixes don't not create anymore complications |
|
Thank you @keyvan-najafy for steps description with fix. Yes, there is some miss mach that should be fixed. |
pplonski
added
help wanted
|
@MrStrzelec, did you get success. I am also getting exactly same error and running on jupyterhub. The web socket connection is successful but worker is queued and getting same error in rest.py in load_notebook method at line no. 32. |
Thanks, it's working for me! I think the main problem is because the server port is not present in the sys.argv called in How the file |
|
Hi @mariliaribeiro, Here is code that creates job for worker mercury/mercury/apps/ws/client.py Lines 137 to 148 in 4a404b3 Here is code that starts worker process based on job params mercury/mercury/apps/ws/tasks.py Lines 34 to 43 in 4a404b3 Sorry, that not everything is working smoothly ... I'm focused right now on ipywidgets support in Mercury. Anyway, have you successfully created app in Mercury? What is your use case? Do you enjoy working with Mercury? |
Hi @pplonski! Thanks a lot! Yeah, I'm enjoy working with mercury. It really helpful when we are building and validating a MVP. I did some tests on Here are some changes. I hope it works for other cases! # mercury/apps/ws/utils.py
import re
...
def get_client_server_url(server_address: str) -> str:
"""
Method to treat server address to WS client job_params.
"""
regex = r"(.*):([0-9].*)"
server_url = server_address
if len(server_address.split('://')[-1].split(":")) == 1:
server_url = server_address + ":9000"
elif re.search(regex, server_address):
server_url = re.sub(regex, "\\1:9000", server_address, 0)
return server_url# mercury/apps/ws/client.py
from apps.ws.utils import get_client_server_url
...
def need_worker(self):
...
job_params = {
....,
"server_url": get_client_server_url(self.server_address)
}With this code we don't need to apply the change in |
/app/ "your file path" |
|
First: I love how easy it is to build apps from notebook, using mercury. The problem I am having is likely related to this thread, hence posting here. The client shows "Waiting for Worker.." and never connects. |
|
This might be an issue with hard coded worker address. |
|
Yes it is. We had to modify that
…On Fri, 1 Mar, 2024, 1:35 pm Piotr, ***@***.***> wrote:
This might be an issue with hard coded worker address.
—
Reply to this email directly, view it on GitHub
<#391 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAYXFJA3GTMRWTQVCVFWQ5DYWAZGJAVCNFSM6AAAAAA6VSAPEOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZSG4YDMOBXHE>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
modified in an upcoming release? |
|
I'm facing the same issue. My understanding is that the root cause is having single
at the moment the method to determine first address determines second address - and simple solution would be to allow overriding it for deployments that need it. Now - the issue is marked |
|
I think the PR #435 address exactly this. Maybe the name |
Hi there!
First of all I'd like to say that I really appreciate your software.
But I'm having trouble with a worker.
Basics:
We don't use any kind of huggingface/docker, so it only works on localhost.
We've run everything as u said in the tutorial, but we're not able to run it correctly.
So we're running server via systemd - python manage.py runserver .
We'll also run worker as systemd - celery -A server worker --loglevel=info -P gevent --concurrency 1 -E
(to run the mercury server no matter is server is restarted)
And yet we're still waiting for Worker.
Do you have any idea what's going on?
Thank you very much!
The text was updated successfully, but these errors were encountered: