Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Server Timeout makes ther listener to crash #7

Open
wants to merge 11 commits into
base: main
Choose a base branch
from

Conversation

XaviArnaus
Copy link
Owner

@XaviArnaus XaviArnaus commented Mar 5, 2024

The mastodon streaming listener may enter into a Server's TimeOut. The wrapper then fails with an Exception and this escalates up until the app. At this point the MastoFeed listener restarts.

More than a big error, it is a normal behavior due to the server timeout. The logging should become less severe.
In this step the idea is to handle the exception and log it as a smaller error, still sending the Janitor notification (to ensure we identify the exception correctly). In a further step it should just avoid using Janitor.

currently, after notifying and logging, it restarts listening. this should continue.

Traceback:

[2024-03-04 13:40:53,107] INFO     masto-feed   MastoFeed listener
[2024-03-04 13:40:53,109] DEBUG    masto-feed   Starting new Mastodon API instance
[2024-03-04 13:40:53,110] DEBUG    masto-feed   Reusing stored User Credentials
[2024-03-04 13:40:53,131] DEBUG    urllib3.connectionpool Starting new HTTPS connection (1): social.arnaus.net:443
[2024-03-04 13:40:53,312] DEBUG    urllib3.connectionpool https://social.arnaus.net:443 "GET /api/v1/instance/ HTTP/1.1" 200 5319
[2024-03-04 13:40:53,357] DEBUG    urllib3.connectionpool https://social.arnaus.net:443 "GET /api/v1/instance/ HTTP/1.1" 200 5319
[2024-03-04 13:40:53,422] DEBUG    urllib3.connectionpool https://social.arnaus.net:443 "GET /api/v1/streaming/user HTTP/1.1" 200 None
[2024-03-05 03:38:08,930] DEBUG    urllib3.connectionpool Starting new HTTP connection (1): dantooine:5000
[2024-03-05 03:38:09,571] DEBUG    urllib3.connectionpool http://dantooine:5000 "POST /message HTTP/1.1" 200 4
[2024-03-05 03:38:09,575] ERROR    masto-feed   exceptions must derive from BaseException
Traceback (most recent call last):
  File "/home/xavier/.cache/pypoetry/virtualenvs/mastofeed-XHz1mY8R-py3.9/lib/python3.9/site-packages/urllib3/response.py", line 737, in _error_catcher
    yield
  File "/home/xavier/.cache/pypoetry/virtualenvs/mastofeed-XHz1mY8R-py3.9/lib/python3.9/site-packages/urllib3/response.py", line 1184, in read_chunked
    self._update_chunk_length()
  File "/home/xavier/.cache/pypoetry/virtualenvs/mastofeed-XHz1mY8R-py3.9/lib/python3.9/site-packages/urllib3/response.py", line 1108, in _update_chunk_length
    line = self._fp.fp.readline()  # type: ignore[union-attr]
  File "/usr/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.9/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.9/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/xavier/.cache/pypoetry/virtualenvs/mastofeed-XHz1mY8R-py3.9/lib/python3.9/site-packages/requests/models.py", line 816, in generate
    yield from self.raw.stream(chunk_size, decode_content=True)
  File "/home/xavier/.cache/pypoetry/virtualenvs/mastofeed-XHz1mY8R-py3.9/lib/python3.9/site-packages/urllib3/response.py", line 1040, in stream
    yield from self.read_chunked(amt, decode_content=decode_content)
  File "/home/xavier/.cache/pypoetry/virtualenvs/mastofeed-XHz1mY8R-py3.9/lib/python3.9/site-packages/urllib3/response.py", line 1213, in read_chunked
    self._original_response.close()
  File "/usr/lib/python3.9/contextlib.py", line 135, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/xavier/.cache/pypoetry/virtualenvs/mastofeed-XHz1mY8R-py3.9/lib/python3.9/site-packages/urllib3/response.py", line 742, in _error_catcher
    raise ReadTimeoutError(self._pool, None, "Read timed out.") from e  # type: ignore[arg-type]
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='social.arnaus.net', port=443): Read timed out.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/xavier/.cache/pypoetry/virtualenvs/mastofeed-XHz1mY8R-py3.9/lib/python3.9/site-packages/mastodon/streaming.py", line 109, in handle_stream
    for chunk in response.iter_content(chunk_size=1):
  File "/home/xavier/.cache/pypoetry/virtualenvs/mastofeed-XHz1mY8R-py3.9/lib/python3.9/site-packages/requests/models.py", line 822, in generate
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='social.arnaus.net', port=443): Read timed out.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/xavier/bots/social-arnaus-feeder/mastofeed/runners/listener.py", line 33, in run
    mastodon_instance.stream_user(mention_listener)
  File "/home/xavier/.cache/pypoetry/virtualenvs/mastofeed-XHz1mY8R-py3.9/lib/python3.9/site-packages/decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
  File "/home/xavier/.cache/pypoetry/virtualenvs/mastofeed-XHz1mY8R-py3.9/lib/python3.9/site-packages/mastodon/utility.py", line 49, in wrapper
    return function(self, *args, **kwargs)
  File "/home/xavier/.cache/pypoetry/virtualenvs/mastofeed-XHz1mY8R-py3.9/lib/python3.9/site-packages/mastodon/streaming_endpoints.py", line 21, in stream_user
    return self.__stream('/api/v1/streaming/user', listener, run_async=run_async, timeout=timeout, reconnect_async=reconnect_async, reconnect_async_wait_sec=reconnect_async_wait_sec)
  File "/home/xavier/.cache/pypoetry/virtualenvs/mastofeed-XHz1mY8R-py3.9/lib/python3.9/site-packages/mastodon/internals.py", line 540, in __stream
    listener.handle_stream(r)
  File "/home/xavier/.cache/pypoetry/virtualenvs/mastofeed-XHz1mY8R-py3.9/lib/python3.9/site-packages/mastodon/streaming.py", line 151, in handle_stream
    six.raise_from(
  File "<string>", line 3, in raise_from
TypeError: exceptions must derive from BaseException
[2024-03-05 03:38:09,581] INFO     masto-feed   MastoFeed listener
[2024-03-05 03:38:09,582] DEBUG    masto-feed   Starting new Mastodon API instance
[2024-03-05 03:38:09,583] DEBUG    masto-feed   Reusing stored User Credentials
[2024-03-05 03:38:09,595] DEBUG    urllib3.connectionpool Starting new HTTPS connection (1): social.arnaus.net:443
[2024-03-05 03:38:09,776] DEBUG    urllib3.connectionpool https://social.arnaus.net:443 "GET /api/v1/instance/ HTTP/1.1" 200 5319
[2024-03-05 03:38:09,828] DEBUG    urllib3.connectionpool https://social.arnaus.net:443 "GET /api/v1/instance/ HTTP/1.1" 200 5319
[2024-03-05 03:38:09,901] DEBUG    urllib3.connectionpool https://social.arnaus.net:443 "GET /api/v1/streaming/user HTTP/1.1" 200 None

@XaviArnaus
Copy link
Owner Author

It's difficult to catch the Exception, as it fails to inherit from BaseException from within Mastodon.py, as explained here:

halcy/Mastodon.py#368

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant