"Error 110 while writing to socket. Connection timed out." With kombu 4.4.0/4.5.0 and redis 3.2.0/3.2.1 #1019

alexandre-paroissien · 2019-03-04T09:38:05Z

I was previously on kombu 4.3.0 and redis 2.10.6, and after upgrading to kombu 4.4.0/4.5.0 and redis 3.2.0/3.2.1 I noticed a new issue on my Django REST endpoints:

Error 110 while writing to socket. Connection timed out.

These endpoints never had any issues before and this issue popped up only once since I upgraded, but happened on both endpoints simultaneously

Edit: Bug still there in most recent versions of the libraries (celery 4.3.0, kombu 4.5.0, redis 3.2.1)
(I also had the bug when just upgrading kombu/redis and keeping celery at 4.2.2)
Bug is not there in celery 4.2.2, kombu 4.3.0, redis 2.10.6

2019-05-21T07:51:02.022118+00:00 app[worker.1]: [2019-05-21 07:51:02,021: ERROR/ForkPoolWorker-5] Connection to Redis lost: Retry (0/20) now.
2019-05-21T07:51:02.024750+00:00 app[worker.1]: [2019-05-21 07:51:02,024: ERROR/ForkPoolWorker-5] Connection to Redis lost: Retry (1/20) in 1.00 second.
2019-05-21T07:51:03.028332+00:00 app[worker.1]: [2019-05-21 07:51:03,028: ERROR/ForkPoolWorker-5] Connection to Redis lost: Retry (2/20) in 1.00 second.
2019-05-21T07:51:04.032513+00:00 app[worker.1]: [2019-05-21 07:51:04,032: ERROR/ForkPoolWorker-5] Connection to Redis lost: Retry (3/20) in 1.00 second.
2019-05-21T07:51:05.037741+00:00 app[worker.1]: [2019-05-21 07:51:05,037: ERROR/ForkPoolWorker-5] Connection to Redis lost: Retry (4/20) in 1.00 second.
2019-05-21T07:51:06.041513+00:00 app[worker.1]: [2019-05-21 07:51:06,041: ERROR/ForkPoolWorker-5] Connection to Redis lost: Retry (5/20) in 1.00 second.
2019-05-21T07:51:07.045367+00:00 app[worker.1]: [2019-05-21 07:51:07,045: ERROR/ForkPoolWorker-5] Connection to Redis lost: Retry (6/20) in 1.00 second.
2019-05-21T07:51:08.048339+00:00 app[worker.1]: [2019-05-21 07:51:08,048: ERROR/ForkPoolWorker-5] Connection to Redis lost: Retry (7/20) in 1.00 second.
2019-05-21T07:51:09.052390+00:00 app[worker.1]: [2019-05-21 07:51:09,052: ERROR/ForkPoolWorker-5] Connection to Redis lost: Retry (8/20) in 1.00 second.

The text was updated successfully, but these errors were encountered:

nyhobbit · 2019-03-12T17:42:39Z

I confirm that I encountered the same issue and also Error 32 (broken pipe) AS PER #1018 after upgrading to the combination of:

celery 4.3.0 RC2
kombu 4.4.0
redis 3.2.0

The errors are intermittent: certain transactions work out and write into redis just fine, others break down. The stack traces when errors occur are identical to the ones from the OP.
I have not been able to uncover any specifics about which transactions are likely to fail and which are likely to work. The entries I write in redis are all very similar, and it looks like sometimes the same entry that had broken down earlier get written just fine later.

The most up-to-date combination of libraries that seems to guarantee the disappearance of these issues is to my knowledge as follows:

celery 4.2.1
kombu 4.3.0
redis 2.10.6

However, I would really like to be able to upgrade to redis >= 3.0 without losing this guarantee. If you could publish a 4.2.x celery version that supports redis 3.x and is stable with respect to #1018 and #1019 it would be greatly appreciated.

alexandre-paroissien · 2019-03-13T10:03:58Z

Here is my environment, exceptions only happen when I switch from kombu 4.3.0 and redis 2.10.6 to kombu 4.4.0 and redis 3.2.0 (The environment and the other libraries remaining unchanged)

Stack:
Ubuntu 18 (Heroku-18)
Python 3.6.8
Redis 3.2.12 (Redis To Go)

Libraries:
Django==2.1.7
djangorestframework==3.9.1
celery==4.2.1
kombu==4.3.0 ok / 4.4.0 not ok
billiard==3.5.0.5
amqp==2.4.2
vine==1.2.0
django-celery-beat==1.4.0
redis==2.10.6 ok / redis 3.2.0 not ok

I hope this can help understanding the exceptions. Is there any other information I could collect or tests I could do?

thedrow · 2019-03-13T10:06:53Z

How about the Redis logs?
Maybe you could also ask upstream?

thedrow · 2019-03-13T10:07:21Z

@nyhobbit I don't think that's possible but we can look into it if you'll open a new issue about it.

alexandre-paroissien · 2019-03-13T10:13:45Z

@thedrow Ok I will check the logs

By upstream you mean python-redis library?

I just checked and it seems you're right! Found the following issues in python-redis:

Sporadic "Connection not ready" exceptions with BlockingConnectionPool since 3.2.0 (22 days ago)
redis/redis-py#1136

3.2.0 Error while reading from socket: ('Connection closed by server.',) (15 days ago)
redis/redis-py#1140

alexandre-paroissien · 2019-03-13T10:23:17Z

These 2 previous issues are there in python-redis 3.2.0

Reverting to 3.1.0 would reintroduce this issue that was fixed in 3.2.0:
redis/redis-py#1127

So maybe the previous python-redis 3.0.1 would be fine?

But it seems that currently kombu is requiring redis >= 3.2.0

thedrow · 2019-03-13T11:36:58Z

Are you using eventlet?

3.0.1 doesn't work since it crashes Celery when forking.

alexandre-paroissien · 2019-03-14T05:44:17Z

I am not using eventlet

I just posted this comment on redis-py redis/redis-py#1140 (comment)

dejlek · 2019-03-22T19:59:49Z

I've noticed similar behaviour as well. I had to fix this by catching the error and reconnecting... It is really annoying. It reminds me of another problem that started few months ago - with connection timeouts. I've reported this to Celery but who knows what is really causing it...

alexandre-paroissien · 2019-05-07T08:42:51Z

Hi @nyhobbit and @dejlek I still have the issue, how about you? Any update?
Maybe you can report your situation there too redis/redis-py#1140

auvipy · 2019-05-07T09:28:28Z

celery 4.3 and latest kombu?

harrybiddle · 2019-05-07T12:05:07Z

Hey @alexandre-paroissien, I'm sorry, I gave up and downgraded to Redis 2.10.6 / Kombu 4.3.0 / a forked Celery 4.2 with Python 3.7 support...!

I also considered doing what @dejlek did; I think if we have to upgrade and the problem is still there that's what I'll have to do too :(.

auvipy · 2019-05-07T13:02:54Z

did you try kombu 4.5 and celery 4.3?

harrybiddle · 2019-05-09T14:23:46Z

I saw this on two stacks: one with

celery 4.3.0rc1
kombu 4.3.0
redis 3.2.0

and another with

celery 4.3.0rc1
kombu 4.4.0
redis 3.2.1

In both cases the downgrade to Celery 4.2 / kombu 4.3.0 / redis 2.10.6 fixed it. I don't know which library was causing the issue!

Unfortunately we don't have a great testing environment for me to try other combinations. It only appeared intermittently on production, and not at all on staging, I guess because the staging traffic was too low. We have a plan to run some load testing on staging. I very much doubt we'll do it any time soon, but if we do get to that and this issue is still open I'll try newer versions of the libraries!

alexandre-paroissien · 2019-05-21T08:00:18Z

I confirm I still encounter this issue in the most recent versions of the libraries
celery 4.3.0, kombu 4.5.0, redis 3.2.1

(I also had the bug when just upgrading kombu/redis and keeping celery at 4.2.1)

I tested in a test app with no traffic apart from me, I launched a simple task manually, first time worked, second time gave the following output (and ending up working)


2019-05-21T07:51:02.022118+00:00 app[worker.1]: [2019-05-21 07:51:02,021: ERROR/ForkPoolWorker-5] Connection to Redis lost: Retry (0/20) now.
2019-05-21T07:51:02.024750+00:00 app[worker.1]: [2019-05-21 07:51:02,024: ERROR/ForkPoolWorker-5] Connection to Redis lost: Retry (1/20) in 1.00 second.
2019-05-21T07:51:03.028332+00:00 app[worker.1]: [2019-05-21 07:51:03,028: ERROR/ForkPoolWorker-5] Connection to Redis lost: Retry (2/20) in 1.00 second.
2019-05-21T07:51:04.032513+00:00 app[worker.1]: [2019-05-21 07:51:04,032: ERROR/ForkPoolWorker-5] Connection to Redis lost: Retry (3/20) in 1.00 second.
2019-05-21T07:51:05.037741+00:00 app[worker.1]: [2019-05-21 07:51:05,037: ERROR/ForkPoolWorker-5] Connection to Redis lost: Retry (4/20) in 1.00 second.
2019-05-21T07:51:06.041513+00:00 app[worker.1]: [2019-05-21 07:51:06,041: ERROR/ForkPoolWorker-5] Connection to Redis lost: Retry (5/20) in 1.00 second.
2019-05-21T07:51:07.045367+00:00 app[worker.1]: [2019-05-21 07:51:07,045: ERROR/ForkPoolWorker-5] Connection to Redis lost: Retry (6/20) in 1.00 second.
2019-05-21T07:51:08.048339+00:00 app[worker.1]: [2019-05-21 07:51:08,048: ERROR/ForkPoolWorker-5] Connection to Redis lost: Retry (7/20) in 1.00 second.
2019-05-21T07:51:09.052390+00:00 app[worker.1]: [2019-05-21 07:51:09,052: ERROR/ForkPoolWorker-5] Connection to Redis lost: Retry (8/20) in 1.00 second.

alexandre-paroissien · 2019-05-22T04:07:36Z

Could it be related to this issue? celery/celery#3932

thedrow · 2019-05-22T08:35:12Z

Your problem appeared while upgrading py-redis so I don't know really.

alexandre-paroissien · 2019-06-20T05:12:44Z

Right, for anyone reading this, it seems the issue appeared from the 3.x branch of redispy indeed (which is required by celery 4.3+ / kombu 4.4+), and the status of the issue is evolving there:

redis/redis-py#1140

Fiftyseventheory · 2019-11-27T17:43:29Z

I'm experiencing this issue as well.
celery==4.3.0
kombu==4.6.4
redis==3.3.8

rotten · 2019-11-27T23:33:49Z

On Nov 27, 2019 12:43 PM, Fiftyseventheory <[email protected]> wrote:I'm experiencing this issue as well. celery==4.3.0 kombu==4.6.4 redis==3.3.8 —You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or unsubscribe. It is definitely better with Celery 4.4.0rc3/4, and slightly better with pyamqp instead of redis. I can't say it is totally solved though. We still see these pop up, just not as often.

auvipy · 2019-11-28T02:05:14Z

thanks for the report. there is an open PR relating to this check #1122 plz

wisefool769 · 2020-01-18T22:21:08Z

I am still seeing this with:
celery==4.4.0
django-redis==4.10.0
kombu==4.6.7
redis==3.3.11

Is there any setting i have to change to pick up the fixes in #1122 ?

jingtt0704 · 2020-01-20T17:09:51Z

I am still seeing this with:
celery==4.4.0
django-redis==4.10.0
kombu==4.6.7
redis==3.3.11

Is there any setting i have to change to pick up the fixes in #1122 ?

I am also still having the issue with the same lib versions

Ashish-Bansal · 2020-01-20T19:26:52Z

@wisefool769 @jingtt0704 This issue will still persist in case celery's event loop is not running.

Ref - #1019 (comment)

sklarsa · 2020-02-21T19:59:06Z

@auvipy can this issue be reopened? I don't think this is fully resolved yet, as I'm experiencing errors similar to many of the recent commenters on this thread

sklarsa · 2020-02-21T20:01:11Z

@Ashish-Bansal Is there any way that I can help fix this in the case you described where celery's event loop is not running?

For clarification, I'm running

celery = "=4.3.0"
kombu = "=4.4.0"
redis = "=2.10.5"

auvipy · 2020-02-26T13:17:36Z

try latest celery with latest kombu

gtalarico · 2020-03-24T16:38:13Z

Sorry to pile up on this, just wanted to report I am having the same error, and I am on the latest releases of celery, kombu and redis.

Traceback

[2020-03-24 16:28:44,787: INFO/MainProcess] mingle: all alone
[2020-03-24 16:30:23,873: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/usr/local/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/usr/local/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 599, in start
    c.loop(*c.loop_args())
  File "/usr/local/lib/python3.7/site-packages/celery/worker/loops.py", line 83, in asynloop
    next(loop)
  File "/usr/local/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 364, in create_loop
    cb(*cbargs)
  File "/usr/local/lib/python3.7/site-packages/kombu/transport/redis.py", line 1088, in on_readable
    self.cycle.on_readable(fileno)
  File "/usr/local/lib/python3.7/site-packages/kombu/transport/redis.py", line 359, in on_readable
    chan.handlers[type]()
  File "/usr/local/lib/python3.7/site-packages/kombu/transport/redis.py", line 693, in _receive
    ret.append(self._receive_one(c))
  File "/usr/local/lib/python3.7/site-packages/kombu/transport/redis.py", line 703, in _receive_one
    response = c.parse_response()
  File "/usr/local/lib/python3.7/site-packages/redis/client.py", line 3453, in parse_response
    response = self._execute(conn, conn.read_response)
  File "/usr/local/lib/python3.7/site-packages/redis/client.py", line 3427, in _execute
    return command(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/redis/connection.py", line 734, in read_response
    response = self._parser.read_response()
  File "/usr/local/lib/python3.7/site-packages/redis/connection.py", line 316, in read_response
    response = self._buffer.readline()
  File "/usr/local/lib/python3.7/site-packages/redis/connection.py", line 248, in readline
    self._read_from_socket()
  File "/usr/local/lib/python3.7/site-packages/redis/connection.py", line 193, in _read_from_socket
    raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.
[2020-03-24 16:30:23,911: INFO/MainProcess] Connected to redis://redis:11535//

celery==4.4.1
kombu==4.6.8
redis==3.4.1

adi- · 2020-04-21T10:33:01Z

Same here. Any fixes on that?

yashu-seth · 2020-10-06T12:10:52Z

Same here.

PaiHsuehChung · 2021-05-13T02:45:17Z

celery==4.4.0
kombu==4.6.11
redis==3.5.3

encounter same issue

hampsterx · 2021-05-13T20:34:01Z

what version of redis are you running? we had a version prior to 5.x and had this issue with following libs noted at the time:

redis 3.2.0
celery 4.3.0

After upgrading we haven't seen the connection time out issue since.

Worth ruling out redis in any case..

PaiHsuehChung · 2021-05-14T05:36:27Z

Redis version is 6.2
So should i downgrading redis version or not?

hampsterx · 2021-05-14T06:36:01Z

nope, was just a idea :(

westonplatter · 2021-05-27T21:29:48Z

After upgrading we haven't seen the connection time out issue since.

@hampsterx what package and version did you upgrade to?

seanquinn · 2021-09-01T12:13:59Z

What's the solution for this - seen recently on:

celery==4.4.0
kombu==4.6.11
redis==3.2.0

matusvalo · 2021-09-11T20:14:00Z

Let me reopen the issue since it seems to be occurring.

KFoxder · 2021-09-11T20:32:35Z

@matusvalo @seanquinn We were able to fix this by doing the following.

You should be able to resolve this by setting the health_check_interval option which was added in kombu version 4.6.8. You will need to specify this via Celery's broker_transport_options option.

broker_transport_options = {
    # Check the health of connections every 5 seconds
    'health_check_interval': 5,
}

auvipy · 2021-09-12T04:59:46Z

@matusvalo @seanquinn We were able to fix this by doing the following.

You should be able to resolve this by setting the health_check_interval option which was added in kombu version 4.6.8. You will need to specify this via Celery's broker_transport_options option.
broker_transport_options = {
    # Check the health of connections every 5 seconds
    'health_check_interval': 5,
}  

is this mentioned in the docs already? if so we can close this

this might solve one of the issues we see when our redis pod restarts celery/kombu#1019

harshita01398 · 2022-06-22T07:06:26Z

Still facing the same issue on

Celery: 4.3.1
Redis: 4.1.2
Kombu: 4.6.11

Even after adding health_check_interval, seeing errors in large numbers.

Have already added below setting
broker_transport_options = {
# Check the health of connections every 5 seconds
'health_check_interval': 5,
}

Do we have a fix in any latest updates?

westonplatter · 2022-06-22T14:00:50Z

Fixed after we upgraded Celery

* Add a health check from celery to redis this solved connection problems between celery and redis for other peeps celery/kombu#1019 * oops had a dev thingy still present in the conf

alexandre-paroissien mentioned this issue Mar 4, 2019

kombu 4.4.0 arguably includes a breaking change and can break user's software. #1016

Closed

thedrow added Component: Redis Broker Issue Type: Bug Report labels Mar 12, 2019

thedrow mentioned this issue Mar 13, 2019

Release version 4.3 of Celery celery/celery#5180

Closed

alexandre-paroissien changed the title ~~"Error 110 while writing to socket. Connection timed out." After upgrading to kombu 4.4.0 and redis 3.0~~ "Error 110 while writing to socket. Connection timed out." After upgrading to kombu 4.4.0 and redis 3.2.0 Mar 13, 2019

jraddaoui mentioned this issue Mar 13, 2019

Problem: Celery worker not working CCA-Public/scope#127

Closed

alexandre-paroissien mentioned this issue Mar 14, 2019

3.2.0 Error while reading from socket: ('Connection closed by server.',) redis/redis-py#1140

Closed

auvipy added this to the 4.5.x Maintenance milestone May 9, 2019

matrixise mentioned this issue May 21, 2019

miss-islington broken again python/miss-islington#228

Closed

alexandre-paroissien changed the title ~~"Error 110 while writing to socket. Connection timed out." After upgrading to kombu 4.4.0 and redis 3.2.0~~ "Error 110 while writing to socket. Connection timed out." With kombu 4.4.0/4.5.0 and redis 3.2.0/3.2.1 May 22, 2019

auvipy closed this as completed Jun 24, 2019

auvipy added the py-redis label Jun 24, 2019

auvipy closed this as completed in #1122 Dec 1, 2019

matusvalo reopened this Sep 11, 2021

alexnsu added a commit to alexnsu/lemur that referenced this issue Mar 29, 2022

configure celery broker to health check redis

0aae528

this might solve one of the issues we see when our redis pod restarts celery/kombu#1019

alexnsu mentioned this issue Mar 29, 2022

configure celery broker to health check redis spotify/lemur#53

Closed

auvipy closed this as completed Apr 23, 2022

auvipy removed the Issue Type: Feature Request label Apr 23, 2022

"Error 110 while writing to socket. Connection timed out." With kombu 4.4.0/4.5.0 and redis 3.2.0/3.2.1 #1019

"Error 110 while writing to socket. Connection timed out." With kombu 4.4.0/4.5.0 and redis 3.2.0/3.2.1 #1019

Comments

alexandre-paroissien commented Mar 4, 2019 • edited

nyhobbit commented Mar 12, 2019

alexandre-paroissien commented Mar 13, 2019

thedrow commented Mar 13, 2019

thedrow commented Mar 13, 2019

alexandre-paroissien commented Mar 13, 2019 • edited

alexandre-paroissien commented Mar 13, 2019 • edited

thedrow commented Mar 13, 2019

alexandre-paroissien commented Mar 14, 2019

dejlek commented Mar 22, 2019

alexandre-paroissien commented May 7, 2019

auvipy commented May 7, 2019

harrybiddle commented May 7, 2019 • edited

auvipy commented May 7, 2019

harrybiddle commented May 9, 2019

alexandre-paroissien commented May 21, 2019 • edited

alexandre-paroissien commented May 22, 2019

thedrow commented May 22, 2019

alexandre-paroissien commented Jun 20, 2019

Fiftyseventheory commented Nov 27, 2019

rotten commented Nov 27, 2019 via email

auvipy commented Nov 28, 2019

wisefool769 commented Jan 18, 2020

jingtt0704 commented Jan 20, 2020

Ashish-Bansal commented Jan 20, 2020

sklarsa commented Feb 21, 2020

sklarsa commented Feb 21, 2020 • edited

auvipy commented Feb 26, 2020

gtalarico commented Mar 24, 2020 • edited

adi- commented Apr 21, 2020

yashu-seth commented Oct 6, 2020

PaiHsuehChung commented May 13, 2021

hampsterx commented May 13, 2021

PaiHsuehChung commented May 14, 2021

hampsterx commented May 14, 2021

westonplatter commented May 27, 2021

seanquinn commented Sep 1, 2021

matusvalo commented Sep 11, 2021 • edited

KFoxder commented Sep 11, 2021

auvipy commented Sep 12, 2021

harshita01398 commented Jun 22, 2022

westonplatter commented Jun 22, 2022

alexandre-paroissien commented Mar 4, 2019 •

edited

alexandre-paroissien commented Mar 13, 2019 •

edited

alexandre-paroissien commented Mar 13, 2019 •

edited

harrybiddle commented May 7, 2019 •

edited

alexandre-paroissien commented May 21, 2019 •

edited

sklarsa commented Feb 21, 2020 •

edited

gtalarico commented Mar 24, 2020 •

edited

matusvalo commented Sep 11, 2021 •

edited