Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GUI getting stuck on startup since the flexpool.io dns went out of service. #17993

Open
QuintLeo opened this issue May 8, 2024 · 10 comments
Open

GUI getting stuck on startup since the flexpool.io dns went out of service. #17993

QuintLeo opened this issue May 8, 2024 · 10 comments
Assignees
@ChiaMineJP
Labels
bug Something isn't working

Comments

@QuintLeo
Copy link

QuintLeo commented May 8, 2024

What happened?

2-3 weeks ago, my GUI started sticking at the "starting services full node" point - the other 3 services seem to start up normally.
I never used database v1 as far as I know of, my database path is already pointing at a v2 database - this was a suggested fix, but as I'm already ON a v2 database it's not my issue.
The error log seems to be pointing at an issue with not finding flexpool.io even though there is NOTHING in my config.yaml that mentions flexpool at all.

The DNS records for flexpool seem to have expired at the time this issue started, and at the time I had been running the GUI just long enough every morning to update it's sync, which was normally happening in a very few minutes and the full node was normally starting up within well under 5 minutes to do the sync.
I do NOT use flexpool, though I think I might have used it for a short while many months to a couple years ago.

I have NOT changed anything in my configuration for quite a long time - months to a couple years.
I have no plots on this machine at all.

Version

2.1.4, 2.2.1, 2.3.0

What platform are you using?

Linux

What ui mode are you using?

GUI

Relevant log output

2024-05-07T12:34:01.213 farmer chia.farmer.farmer         : ERROR    Exception in GET /pool_info https://xch-us.flexpool.io, Cannot connect to host xch-us.flexpool.io:443 ssl:<ssl.SSLContext object at 0x7c19400e74c0> [Name or service not known]
2024-05-07T12:34:01.694 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: 4aa763c9823dea2f9fbf83cd02e99b5518d01df87617074f737abf99b0af115d new-peak-prev: eba3caec0cc3f444f18163026aeb38d31e07168b87cdcf09ef7d47640fa3e210 coins: not set
2024-05-07T12:34:05.855 full_node chia.server.start_service: ERROR    fatal uncaught exception: OSError: [Errno 98] error while attempting to bind on address ('0.0.0.0', 8444): address already in use
Traceback (most recent call last):
  File "chia/util/log_exceptions.py", line 20, in log_exceptions
  File "chia/server/start_service.py", line 323, in async_run
  File "asyncio/runners.py", line 44, in run
  File "asyncio/base_events.py", line 649, in run_until_complete
  File "chia/server/start_full_node.py", line 85, in async_main
  File "chia/server/start_service.py", line 197, in run
  File "contextlib.py", line 199, in __aenter__
  File "chia/server/start_service.py", line 220, in manage
  File "chia/server/server.py", line 288, in start
  File "chia/util/network.py", line 99, in create
  File "chia/util/network.py", line 111, in start
  File "aiohttp/web_runner.py", line 119, in start
  File "chia/server/chia_policy.py", line 236, in create_server
  File "chia/server/chia_policy.py", line 199, in _chia_create_server
  File "asyncio/base_events.py", line 1519, in create_server
OSError: [Errno 98] error while attempting to bind on address ('0.0.0.0', 8444): address already in use

2024-05-07T12:36:03.631 farmer chia.farmer.farmer         : ERROR    Exception in GET /pool_info https://xch-us.flexpool.io, Cannot connect to host xch-us.flexpool.io:443 ssl:<ssl.SSLContext object at 0x7c19400e4f40> [Name or service not known]
2024-05-07T12:38:06.035 farmer chia.farmer.farmer         : ERROR    Exception in GET /pool_info https://xch-us.flexpool.io, Cannot connect to host xch-us.flexpool.io:443 ssl:<ssl.SSLContext object at 0x7c193ff7a040> [Name or service not known]
2024-05-07T12:38:57.536 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2024-05-07T12:38:57.537 wallet wallet_server              : WARNING  Banning 79.174.52.102 for 120 seconds
2024-05-07T12:40:08.460 farmer chia.farmer.farmer         : ERROR    Exception in GET /pool_info https://xch-us.flexpool.io, Cannot connect to host xch-us.flexpool.io:443 ssl:<ssl.SSLContext object at 0x7c193ff79840> [Name or service not known]
2024-05-07T12:42:10.860 farmer chia.farmer.farmer         : ERROR    Exception in GET /pool_info https://xch-us.flexpool.io, Cannot connect to host xch-us.flexpool.io:443 ssl:<ssl.SSLContext object at 0x7c193ff795c0> [Name or service not known]
2024-05-07T12:44:13.326 farmer chia.farmer.farmer         : ERROR    Exception in GET /pool_info https://xch-us.flexpool.io, Cannot connect to host xch-us.flexpool.io:443 ssl:<ssl.SSLContext object at 0x7c193ff79140> [Name or service not known]
2024-05-07T12:44:13.327 farmer chia.farmer.farmer         : WARNING  No pool specific authentication_token_timeout has been set for 06a59ad8e9af0cb345ed28c827551c364863d645334e3fc85421d522f0cca422, check communication with the pool.
2024-05-07T12:46:15.750 farmer chia.farmer.farmer         : ERROR    Exception in GET /pool_info https://xch-us.flexpool.io, Cannot connect to host xch-us.flexpool.io:443 ssl:<ssl.SSLContext object at 0x7c193ff79840> [Name or service not known]
2024-05-07T12:47:35.266 wallet chia.wallet.wallet_node    : WARNING  Peer PeerInfo(_ip=IPv4Address('158.174.215.224'), _port=8444) did not respond in time.
2024-05-07T12:47:35.267 wallet wallet_server              : WARNING  Banning 158.174.215.224 for 120 seconds
2024-05-07T12:48:18.188 farmer chia.farmer.farmer         : ERROR    Exception in GET /pool_info https://xch-us.flexpool.io, Cannot connect to host xch-us.flexpool.io:443 ssl:<ssl.SSLContext object at 0x7c193ff795c0> [Name or service not known]
2024-05-07T12:50:20.585 farmer chia.farmer.farmer         : ERROR    Exception in GET /pool_info https://xch-us.flexpool.io, Cannot connect to host xch-us.flexpool.io:443 ssl:<ssl.SSLContext object at 0x7c193ff79140> [Name or service not known]
2024-05-07T12:52:23.005 farmer chia.farmer.farmer         : ERROR    Exception in GET /pool_info https://xch-us.flexpool.io, Cannot connect to host xch-us.flexpool.io:443 ssl:<ssl.SSLContext object at 0x7c193ff7b1c0> [Name or service not known]
2024-05-07T12:54:25.427 farmer chia.farmer.farmer         : ERROR    Exception in GET /pool_info https://xch-us.flexpool.io, Cannot connect to host xch-us.flexpool.io:443 ssl:<ssl.SSLContext object at 0x7c193ff7a040> [Name or service not known]
2024-05-07T12:54:25.427 farmer chia.farmer.farmer         : WARNING  No pool specific authentication_token_timeout has been set for 06a59ad8e9af0cb345ed28c827551c364863d645334e3fc85421d522f0cca422, check communication with the pool.
2024-05-07T12:56:27.846 farmer chia.farmer.farmer         : ERROR    Exception in GET /pool_info https://xch-us.flexpool.io, Cannot connect to host xch-us.flexpool.io:443 ssl:<ssl.SSLContext object at 0x7c193ff79140> [Name or service not known]
2024-05-07T12:58:30.234 farmer chia.farmer.farmer         : ERROR    Exception in GET /pool_info https://xch-us.flexpool.io, Cannot connect to host xch-us.flexpool.io:443 ssl:<ssl.SSLContext object at 0x7c193ff7b1c0> [Name or service not known]
2024-05-07T13:00:32.677 farmer chia.farmer.farmer         : ERROR    Exception in GET /pool_info https://xch-us.flexpool.io, Cannot connect to host xch-us.flexpool.io:443 ssl:<ssl.SSLContext object at 0x7c193ff7a040> [Name or service not known]
2024-05-07T13:02:35.096 farmer chia.farmer.farmer         : ERROR    Exception in GET /pool_info https://xch-us.flexpool.io, Cannot connect to host xch-us.flexpool.io:443 ssl:<ssl.SSLContext object at 0x7c193ff79140> [Name or service not known]
2024-05-07T13:04:37.506 farmer chia.farmer.farmer         : ERROR    Exception in GET /pool_info https://xch-us.flexpool.io, Cannot connect to host xch-us.flexpool.io:443 ssl:<ssl.SSLContext object at 0x7c193ff7b1c0> [Name or service not known]
2024-05-07T13:04:37.506 farmer chia.farmer.farmer         : WARNING  No pool specific authentication_token_timeout has been set for 06a59ad8e9af0cb345ed28c827551c364863d645334e3fc85421d522f0cca422, check communication with the pool.
2024-05-07T13:06:39.922 farmer chia.farmer.farmer         : ERROR    Exception in GET /pool_info https://xch-us.flexpool.io, Cannot connect to host xch-us.flexpool.io:443 ssl:<ssl.SSLContext object at 0x7c193ff7a040> [Name or service not known]
2024-05-07T13:07:31.926 wallet chia.wallet.wallet_node    : WARNING  Peer PeerInfo(_ip=IPv4Address('32.219.186.109'), _port=8444) did not respond in time.
2024-05-07T13:07:31.927 wallet wallet_server              : WARNING  Banning 32.219.186.109 for 120 seconds
2024-05-07T13:07:31.927 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2024-05-07T13:07:31.927 wallet wallet_server              : WARNING  Banning 24.6.202.14 for 120 seconds
2024-05-07T13:07:31.928 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2024-05-07T13:07:31.928 wallet wallet_server              : WARNING  Banning 95.70.208.109 for 120 seconds
2024-05-07T13:07:31.928 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2024-05-07T13:07:31.928 wallet wallet_server              : WARNING  Banning 24.6.202.14 for 120 seconds
2024-05-07T13:07:31.928 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2024-05-07T13:07:31.929 wallet wallet_server              : WARNING  Banning 95.70.208.109 for 120 seconds
2024-05-07T13:07:31.929 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2024-05-07T13:07:31.929 wallet wallet_server              : WARNING  Banning 95.70.208.109 for 120 seconds
2024-05-07T13:07:31.929 wallet chia.wallet.wallet_node    : WARNING  Peer None did not respond in time.
2024-05-07T13:07:31.929 wallet wallet_server              : WARNING  Banning 24.6.202.14 for 120 seconds
2024-05-07T13:08:42.365 farmer chia.farmer.farmer         : ERROR    Exception in GET /pool_info https://xch-us.flexpool.io, Cannot connect to host xch-us.flexpool.io:443 ssl:<ssl.SSLContext object at 0x7c193ff7a4c0> [Name or service not known]
2024-05-07T13:10:44.785 farmer chia.farmer.farmer         : ERROR    Exception in GET /pool_info https://xch-us.flexpool.io, Cannot connect to host xch-us.flexpool.io:443 ssl:<ssl.SSLContext object at 0x7c193ff78a40> [Name or service not known]
2024-05-07T13:12:47.189 farmer chia.farmer.farmer         : ERROR    Exception in GET /pool_info https://xch-us.flexpool.io, Cannot connect to host xch-us.flexpool.io:443 ssl:<ssl.SSLContext object at 0x7c193ff7a040> [Name or service not known]
2024-05-07T13:14:49.577 farmer chia.farmer.farmer         : ERROR    Exception in GET /pool_info https://xch-us.flexpool.io, Cannot connect to host xch-us.flexpool.io:443 ssl:<ssl.SSLContext object at 0x7c193ff7a4c0> [Name or service not known]
2024-05-07T13:14:49.577 farmer chia.farmer.farmer         : WARNING  No pool specific authentication_token_timeout has been set for 06a59ad8e9af0cb345ed28c827551c364863d645334e3fc85421d522f0cca422, check communication with the pool.
2024-05-07T13:16:52.036 farmer chia.farmer.farmer         : ERROR    Exception in GET /pool_info https://xch-us.flexpool.io, Cannot connect to host xch-us.flexpool.io:443 ssl:<ssl.SSLContext object at 0x7c19400e7c40> [Name or service not known]
2024-05-07T13:18:54.462 farmer chia.farmer.farmer         : ERROR    Exception in GET /pool_info https://xch-us.flexpool.io, Cannot connect to host xch-us.flexpool.io:443 ssl:<ssl.SSLContext object at 0x7c19400e7c40> [Name or service not known]
2024-05-07T13:19:08.820 wallet chia.wallet.wallet_node    : WARNING  Peer PeerInfo(_ip=IPv4Address('60.240.184.81'), _port=8444) did not respond in time.
2024-05-07T13:19:08.821 wallet wallet_server              : WARNING  Banning 60.240.184.81 for 120 seconds
2024-05-08T14:18:16.788 farmer chia.farmer.farmer         : ERROR    Exception in GET /pool_info https://xch-us.flexpool.io, Cannot connect to host xch-us.flexpool.io:443 ssl:<ssl.SSLContext object at 0x77393e8cdf40> [Name or service not known]
2024-05-08T14:18:16.961 full_node chia.full_node.mempool_manager: WARNING  updating the mempool using the slow-path. peak: 4aa763c9823dea2f9fbf83cd02e99b5518d01df87617074f737abf99b0af115d new-peak-prev: eba3caec0cc3f444f18163026aeb38d31e07168b87cdcf09ef7d47640fa3e210 coins: not set
2024-05-08T14:18:21.123 full_node chia.server.start_service: ERROR    fatal uncaught exception: OSError: [Errno 98] error while attempting to bind on address ('::', 8444, 0, 0): address already in use
Traceback (most recent call last):
  File "chia/util/log_exceptions.py", line 20, in log_exceptions
  File "chia/server/start_service.py", line 323, in async_run
  File "asyncio/runners.py", line 44, in run
  File "asyncio/base_events.py", line 649, in run_until_complete
  File "chia/server/start_full_node.py", line 85, in async_main
  File "chia/server/start_service.py", line 197, in run
  File "contextlib.py", line 199, in __aenter__
  File "chia/server/start_service.py", line 220, in manage
  File "chia/server/server.py", line 288, in start
  File "chia/util/network.py", line 99, in create
  File "chia/util/network.py", line 111, in start
  File "aiohttp/web_runner.py", line 119, in start
  File "chia/server/chia_policy.py", line 236, in create_server
  File "chia/server/chia_policy.py", line 199, in _chia_create_server
  File "asyncio/base_events.py", line 1519, in create_server
OSError: [Errno 98] error while attempting to bind on address ('::', 8444, 0, 0): address already in use
@QuintLeo QuintLeo added the bug Something isn't working label May 8, 2024
@OverActiveBladderSystem
Copy link 

2024-05-07T12:34:05.855 full_node chia.server.start_service: ERROR    fatal uncaught exception: OSError: [Errno 98] error while attempting to bind on address ('0.0.0.0', 8444): address already in use

When looking at the quoted line above it suggests to me the "full node" had trouble claiming port 8444 from the OS to use for itself, as if the system reserved it for something else, or something else is currently running on it?

Are you able to look and see if you have other processes running on the machine? Something that wasn't stopped previously? Something that starts up automatically? Do any other processes have control of port 8444? Perhaps signed on as another user who launched something on that port and now you're on another user account trying the same? Since I am naive to linux does the user account being used have permission to utilize that port?

As far as I understand the Chia services, the "farmer" can have its errors and issues without impacting the "full node" service from running, but in this case the "full node" service wasn't able to get going because of the OS port issue? I would expect the "farmer" service to fail if it was unable to resolve the domain name, not the "full node" service. (The "farmer" uses port 8447 and the "full node" uses 8444.)

@QuintLeo
Copy link
Author

QuintLeo commented May 9, 2024

The only thing that has ever used port 8444 on that machine is the Chia full node.
There are NO other users on the machine. There aren't any other accounts to sign in WITH.
Nothing changed in the configuration of the machine itself to cause any issues when I started seeing this issue, and 2.1.4 WAS working before I started seeing this issue for many months to a year.

@QuintLeo
Copy link
Author

Additional info I just figured out.
The CLI WILL start the full node.
It's specifically the GUI that is hanging.

@wjblanke
Copy link
Contributor

Try rebooting before running anything to remove the possibility of zombies.

You can go into config.yaml and see if there are any references to flexpool. It may be best to delete config.yaml and have chia recreate it by resyncing your wallet

@QuintLeo
Copy link
Author

I've done the reboot then run immediately before I run anything else a few times now.
No change.
There are no references to flexpool in my config.yaml - I forgot to mention that in the original post but I checked that a couple weeks before I posted this bug report.
It does not seem to be the wallet that's hanging, or I couldn't get into that via the CLI.
There doesn't seem to be any way to start the full node independently via the CLI, though there is an "all" option in the CLI that seems to work.

@QuintLeo
Copy link
Author

2 weeks and hasn't even been assigned?
Sad.

@wjblanke
Copy link
Contributor

This is the interesting part

The CLI WILL start the full node.
It's specifically the GUI that is hanging.

We did change how the GUI launches services recently. Izumi can u check on this?

@wjblanke
Copy link
Contributor

Can we get a new log file after the reboot? That should clear the 8444 error.

https://docs.chia.net/installation/?_highlight=install#cli
Also you can try running without the GUI

chia start node

don't use "all" as that will start a timelord

@QuintLeo
Copy link
Author

I started the client for the first time in perhaps 2 weeks today - it started up the GUI for the first time in nearly a month.
I have NO IDEA why it wasn't opening for a while, as I've not changed anything.

2024-06-12T13:31:23.701 farmer chia.farmer.farmer : ERROR Exception in GET /pool_info https://xch-us.flexpool.io, Cannot connect to host xch-us.flexpool.io:443 ssl:<ssl.SSLContext object at 0x7fd8df9734c0> [Name or service not known]
2024-06-12T13:31:23.792 full_node chia.full_node.mempool_manager: WARNING updating the mempool using the slow-path. peak: c651a85ab23412fcbb7436d897a4a53d81beac0eea591c041228c81ab986dde0 new-peak-prev: 8cc7cd484a120d87a25ffd93ec32f0237d6ea69106c7be40b34ab9877cd3149e coins: not set
2024-06-12T13:33:26.201 farmer chia.farmer.farmer : ERROR Exception in GET /pool_info https://xch-us.flexpool.io, Cannot connect to host xch-us.flexpool.io:443 ssl:<ssl.SSLContext object at 0x7fd8dfadfc40> [Name or service not known]

@ChiaMineJP
Copy link
Contributor

I investigated source code and here is the summary.

FullNode won't stop if it fails to GET /pool_info ...

farmer chia.farmer.farmer : ERROR Exception in GET /pool_info ... doesn't shutdown fullNode nor stop executing code. (Actually this should be labeled as WARNING)
A farmer just retries the GET request periodically.
So this is not relevant to the issue where fullNode is not launching.

The pool url is loaded only from config.yaml

If you're sure that flexpool is not in your config.yaml, then it is loaded from another config.yaml you haven't checked yet. I suspect unexpected value was set to $CHIA_ROOT env var or sometimes the env var is loaded and sometimes not.
Maybe your .bashrc or .profile or something like those scripts failed before setting CHIA_ROOT and sometimes succeeded.

image

The log updating the mempool using the slow-path indicates fullNode was already running

This is only logged when new_peak (the latest block) is sent from other fullNodes and local fullNodes is accepting it. This means a fullNode was already running and received peak block from external fullNodes.
So the subsequent error (OSError: [Errno 98] error while attempting to bind on address ('::', 8444, 0, 0): address already in use) indicates that something attepted to launch another fullNode even if it was already running.

If you are using Chia >= 2.3.0, even if a fullNode is already running, the GUI just skip to launch another fullNode.

So I'm suspecting that you tried to lauch the GUI whose version is less than 2.3.0 when a fullNode was already running.

More info is required for further investigation

I need more info to proceed my investigation.

  • What is the actual Linux OS you used? (Ubuntu 24.04? Fedora 41? or ArcLInux 2022.12.02?)
  • Did you launch the GUI via source install? or package install?
  • What does chia version CLI command return and what the GUI's version dialog shows?

image

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ChiaMineJP ChiaMineJP

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@wjblanke @QuintLeo @OverActiveBladderSystem @ChiaMineJP