Error 502 was reported on the interface

[Zian502]

Self-Hosted Version

23.12.1

CPU Architecture

x86_64

Docker Version

24.0.6

Docker Compose Version

2.21.0

Steps to Reproduce

At first it's fine, but after a while it's not

Expected Result

The error information can be collected normally

Actual Result

sentry-self-hosted-snuba-profiling-profiles-consumer-1 | 2024-04-06 07:01:34,751 Partitions to revoke: [Partition(topic=Topic(name='processed-profiles'), index=0)]
sentry-self-hosted-snuba-profiling-profiles-consumer-1 | 2024-04-06 07:01:34,752 Closing <arroyo.processing.strategies.guard.StrategyGuard object at 0x7f6188ba74c0>...
sentry-self-hosted-snuba-profiling-profiles-consumer-1 | 2024-04-06 07:01:34,752 Waiting for <arroyo.processing.strategies.guard.StrategyGuard object at 0x7f6188ba74c0> to exit...
sentry-self-hosted-snuba-profiling-profiles-consumer-1 | 2024-04-06 07:01:34,752 <arroyo.processing.strategies.guard.StrategyGuard object at 0x7f6188ba74c0> exited successfully, releasing assignment.
sentry-self-hosted-snuba-profiling-profiles-consumer-1 | 2024-04-06 07:01:34,752 Partition revocation complete.
sentry-self-hosted-snuba-profiling-profiles-consumer-1 | 2024-04-06 07:01:35,245 New partitions assigned: {Partition(topic=Topic(name='processed-profiles'), index=0): 0}
sentry-self-hosted-snuba-profiling-profiles-consumer-1 | 2024-04-06 07:01:46,755 Partitions to revoke: [Partition(topic=Topic(name='processed-profiles'), index=0)]
sentry-self-hosted-snuba-profiling-profiles-consumer-1 | 2024-04-06 07:01:46,755 Closing <arroyo.processing.strategies.guard.StrategyGuard object at 0x7f6188ba74c0>...
sentry-self-hosted-snuba-profiling-profiles-consumer-1 | 2024-04-06 07:01:46,755 Waiting for <arroyo.processing.strategies.guard.StrategyGuard object at 0x7f6188ba74c0> to exit...
sentry-self-hosted-snuba-profiling-profiles-consumer-1 | 2024-04-06 07:01:46,756 <arroyo.processing.strategies.guard.StrategyGuard object at 0x7f6188ba74c0> exited successfully, releasing assignment.
sentry-self-hosted-snuba-profiling-profiles-consumer-1 | 2024-04-06 07:01:46,756 Partition revocation complete.
sentry-self-hosted-snuba-profiling-profiles-consumer-1 | 2024-04-06 07:01:46,771 New partitions assigned: {Partition(topic=Topic(name='processed-profiles'), index=0): 0}
sentry-self-hosted-snuba-outcomes-consumer-1 | 2024-04-06 07:01:32,721 Caught exception, shutting down...
sentry-self-hosted-snuba-outcomes-consumer-1 | Traceback (most recent call last):
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
sentry-self-hosted-snuba-outcomes-consumer-1 | httplib_response = self._make_request(
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request
sentry-self-hosted-snuba-outcomes-consumer-1 | six.raise_from(e, None)
sentry-self-hosted-snuba-outcomes-consumer-1 | File "", line 3, in raise_from
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request
sentry-self-hosted-snuba-outcomes-consumer-1 | httplib_response = conn.getresponse()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/sentry_sdk/integrations/stdlib.py", line 126, in getresponse
sentry-self-hosted-snuba-outcomes-consumer-1 | rv = real_getresponse(self, *args, **kwargs)
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/http/client.py", line 1348, in getresponse
sentry-self-hosted-snuba-outcomes-consumer-1 | response.begin()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/http/client.py", line 316, in begin
sentry-self-hosted-snuba-outcomes-consumer-1 | version, status, reason = self._read_status()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/http/client.py", line 277, in _read_status
sentry-self-hosted-snuba-outcomes-consumer-1 | line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
sentry-self-hosted-snuba-outcomes-consumer-1 | return self._sock.recv_into(b)
sentry-self-hosted-snuba-outcomes-consumer-1 | ConnectionResetError: [Errno 104] Connection reset by peer
sentry-self-hosted-snuba-outcomes-consumer-1 |
sentry-self-hosted-snuba-outcomes-consumer-1 | During handling of the above exception, another exception occurred:
sentry-self-hosted-snuba-outcomes-consumer-1 |
sentry-self-hosted-snuba-outcomes-consumer-1 | Traceback (most recent call last):
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 319, in run
sentry-self-hosted-snuba-outcomes-consumer-1 | self._run_once()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 393, in _run_once
sentry-self-hosted-snuba-outcomes-consumer-1 | self.__processing_strategy.poll()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/strategies/guard.py", line 101, in poll
sentry-self-hosted-snuba-outcomes-consumer-1 | self.__inner_strategy.poll()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/strategies/run_task.py", line 55, in poll
sentry-self-hosted-snuba-outcomes-consumer-1 | self.__next_step.poll()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/strategies/guard.py", line 37, in poll
sentry-self-hosted-snuba-outcomes-consumer-1 | self.__next_step.poll()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/strategies/reduce.py", line 168, in poll
sentry-self-hosted-snuba-outcomes-consumer-1 | self.__next_step.poll()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/strategies/run_task_in_threads.py", line 107, in poll
sentry-self-hosted-snuba-outcomes-consumer-1 | result = future.result()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 437, in result
sentry-self-hosted-snuba-outcomes-consumer-1 | return self.__get_result()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
sentry-self-hosted-snuba-outcomes-consumer-1 | raise self._exception
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
sentry-self-hosted-snuba-outcomes-consumer-1 | result = self.fn(*self.args, **self.kwargs)
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 787, in urlopen
sentry-self-hosted-snuba-outcomes-consumer-1 | retries = retries.increment(
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 550, in increment
sentry-self-hosted-snuba-outcomes-consumer-1 | raise six.reraise(type(error), error, _stacktrace)
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 769, in reraise
sentry-self-hosted-snuba-outcomes-consumer-1 | raise value.with_traceback(tb)
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
sentry-self-hosted-snuba-outcomes-consumer-1 | httplib_response = self._make_request(
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request
sentry-self-hosted-snuba-outcomes-consumer-1 | six.raise_from(e, None)
sentry-self-hosted-snuba-outcomes-consumer-1 | File "", line 3, in raise_from
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request
sentry-self-hosted-snuba-outcomes-consumer-1 | httplib_response = conn.getresponse()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/sentry_sdk/integrations/stdlib.py", line 126, in getresponse
sentry-self-hosted-snuba-outcomes-consumer-1 | rv = real_getresponse(self, *args, **kwargs)
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/http/client.py", line 1348, in getresponse
sentry-self-hosted-snuba-outcomes-consumer-1 | response.begin()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/http/client.py", line 316, in begin
sentry-self-hosted-snuba-outcomes-consumer-1 | version, status, reason = self._read_status()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/http/client.py", line 277, in _read_status
sentry-self-hosted-snuba-outcomes-consumer-1 | line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
sentry-self-hosted-snuba-outcomes-consumer-1 | return self._sock.recv_into(b)
sentry-self-hosted-snuba-outcomes-consumer-1 | urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
sentry-self-hosted-snuba-outcomes-consumer-1 | 2024-04-06 07:01:33,108 Closing <arroyo.backends.kafka.consumer.KafkaConsumer object at 0x7f5b4df4d130>...
sentry-self-hosted-snuba-outcomes-consumer-1 | 2024-04-06 07:01:33,108 Partitions to revoke: [Partition(topic=Topic(name='outcomes'), index=0)]
sentry-self-hosted-snuba-outcomes-consumer-1 | 2024-04-06 07:01:33,109 Partition revocation complete.
sentry-self-hosted-snuba-outcomes-consumer-1 | 2024-04-06 07:01:33,123 Processor terminated
sentry-self-hosted-snuba-outcomes-consumer-1 | Traceback (most recent call last):
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
sentry-self-hosted-snuba-outcomes-consumer-1 | httplib_response = self._make_request(
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request
sentry-self-hosted-snuba-outcomes-consumer-1 | six.raise_from(e, None)
sentry-self-hosted-snuba-outcomes-consumer-1 | File "", line 3, in raise_from
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request
sentry-self-hosted-snuba-outcomes-consumer-1 | httplib_response = conn.getresponse()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/sentry_sdk/integrations/stdlib.py", line 126, in getresponse
sentry-self-hosted-snuba-outcomes-consumer-1 | rv = real_getresponse(self, *args, **kwargs)
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/http/client.py", line 1348, in getresponse
sentry-self-hosted-snuba-outcomes-consumer-1 | response.begin()
sentry-self-hosted-snuba-api-1 | 2024-04-06 04:06:29,180 Received referrer: eventstore.backend.get_event_by_id_nodestore
sentry-self-hosted-snuba-api-1 | 2024-04-06 04:21:23,636 Received referrer: eventstore.backend.get_event_by_id_nodestore
sentry-self-hosted-snuba-api-1 | 2024-04-06 15:00:00,070 Received referrer: dynamic_sampling.counters.get_org_transaction_volumes
sentry-self-hosted-snuba-api-1 | 2024-04-07 05:40:00,148 Received referrer: dynamic_sampling.counters.fetch_projects_with_count_per_transaction_volumes
sentry-self-hosted-snuba-api-1 | ...The work of process 38 is done. Seeya!
sentry-self-hosted-snuba-api-1 | 2024-04-07 11:30:00,236 Flushing and closing cogs recorder if it exists...
sentry-self-hosted-snuba-api-1 | worker 1 killed successfully (pid: 38)
sentry-self-hosted-snuba-api-1 | Respawned uWSGI worker 1 (new pid: 43)
sentry-self-hosted-snuba-api-1 | 2024-04-07 11:30:02,213 Initializing Snuba...
sentry-self-hosted-snuba-api-1 | WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x7f0a96c5a000 pid: 43 (default app)
sentry-self-hosted-snuba-api-1 | 2024-04-07 21:00:00,081 Received referrer: dynamic_sampling.counters.fetch_projects_with_count_per_transaction_volumes
sentry-self-hosted-snuba-api-1 | 2024-04-12 21:00:00,187 Received referrer: dynamic_sampling.counters.fetch_projects_with_count_per_transaction_volumes
sentry-self-hosted-snuba-api-1 | 2024-04-13 23:20:00,152 Received referrer: dynamic_sampling.counters.fetch_projects_with_count_per_transaction_volumes
sentry-self-hosted-snuba-api-1 | 2024-04-15 04:50:00,144 Received referrer: dynamic_sampling.counters.fetch_projects_with_count_per_transaction_volumes
sentry-self-hosted-snuba-api-1 | 2024-04-16 20:30:00,128 Received referrer: dynamic_sampling.counters.fetch_projects_with_count_per_transaction_volumes
sentry-self-hosted-snuba-api-1 | 2024-04-19 03:00:00,055 Received referrer: dynamic_sampling.counters.get_org_transaction_volumes
sentry-self-hosted-snuba-api-1 | 2024-04-19 18:40:00,097 Received referrer: dynamic_sampling.counters.fetch_projects_with_count_per_transaction_volumes
sentry-self-hosted-snuba-api-1 | ...The work of process 43 is done. Seeya!
sentry-self-hosted-snuba-api-1 | 2024-04-20 19:40:00,119 Flushing and closing cogs recorder if it exists...
sentry-self-hosted-snuba-api-1 | worker 1 killed successfully (pid: 43)
sentry-self-hosted-snuba-api-1 | Respawned uWSGI worker 1 (new pid: 48)
sentry-self-hosted-snuba-api-1 | 2024-04-20 19:40:01,691 Initializing Snuba...
sentry-self-hosted-snuba-api-1 | WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x7f0a96c5a000 pid: 48 (default app)
sentry-self-hosted-snuba-api-1 | 2024-04-21 00:50:00,184 Received referrer: dynamic_sampling.counters.fetch_projects_with_count_per_transaction_volumes
sentry-self-hosted-snuba-api-1 | 2024-04-21 11:00:00,160 Received referrer: dynamic_sampling.counters.fetch_projects_with_count_per_transaction_volumes
sentry-self-hosted-snuba-api-1 | 2024-04-22 17:00:00,303 Received referrer: dynamic_sampling.counters.fetch_projects_with_count_per_transaction_volumes
sentry-self-hosted-snuba-api-1 | 2024-04-22 20:30:00,298 Received referrer: dynamic_sampling.counters.get_org_transaction_volumes
sentry-self-hosted-snuba-api-1 | 2024-04-23 21:10:00,147 Received referrer: dynamic_sampling.counters.fetch_projects_with_count_per_transaction_volumes
sentry-self-hosted-snuba-api-1 | 2024-04-25 06:40:00,117 Received referrer: dynamic_sampling.counters.fetch_projects_with_count_per_transaction_volumes
sentry-self-hosted-snuba-api-1 | 2024-04-26 17:20:00,178 Received referrer: dynamic_sampling.counters.fetch_projects_with_count_per_transaction_volumes
sentry-self-hosted-snuba-api-1 | 2024-05-01 03:00:00,204 Received referrer: dynamic_sampling.counters.get_org_transaction_volumes
sentry-self-hosted-snuba-api-1 | 2024-05-01 12:50:00,092 Received referrer: dynamic_sampling.counters.fetch_projects_with_count_per_transaction_volumes
sentry-self-hosted-snuba-api-1 | 2024-05-03 20:50:00,078 Received referrer: dynamic_sampling.counters.fetch_projects_with_count_per_transaction_volumes
sentry-self-hosted-snuba-api-1 | 2024-05-06 15:30:00,226 Received referrer: dynamic_sampling.counters.get_org_transaction_volumes
sentry-self-hosted-snuba-api-1 | ...The work of process 48 is done. Seeya!
sentry-self-hosted-snuba-api-1 | 2024-05-06 18:30:00,103 Flushing and closing cogs recorder if it exists...
sentry-self-hosted-snuba-api-1 | worker 1 killed successfully (pid: 48)
sentry-self-hosted-snuba-api-1 | Respawned uWSGI worker 1 (new pid: 53)
sentry-self-hosted-snuba-api-1 | 2024-05-06 18:30:01,910 Initializing Snuba...
sentry-self-hosted-snuba-api-1 | WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x7f0a96c5a000 pid: 53 (default app)
sentry-self-hosted-snuba-consumer-1 | 2024-04-06 07:01:32,870 Caught exception, shutting down...
sentry-self-hosted-snuba-consumer-1 | Traceback (most recent call last):
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
sentry-self-hosted-snuba-consumer-1 | httplib_response = self._make_request(
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request
sentry-self-hosted-snuba-consumer-1 | six.raise_from(e, None)
sentry-self-hosted-snuba-consumer-1 | File "", line 3, in raise_from
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request
sentry-self-hosted-snuba-consumer-1 | httplib_response = conn.getresponse()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/sentry_sdk/integrations/stdlib.py", line 126, in getresponse
sentry-self-hosted-snuba-consumer-1 | rv = real_getresponse(self, *args, **kwargs)
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/http/client.py", line 1348, in getresponse
sentry-self-hosted-snuba-consumer-1 | response.begin()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/http/client.py", line 316, in begin
sentry-self-hosted-snuba-consumer-1 | version, status, reason = self._read_status()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/http/client.py", line 277, in _read_status
sentry-self-hosted-snuba-consumer-1 | line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
sentry-self-hosted-snuba-consumer-1 | return self._sock.recv_into(b)
sentry-self-hosted-snuba-consumer-1 | ConnectionResetError: [Errno 104] Connection reset by peer
sentry-self-hosted-snuba-consumer-1 |
sentry-self-hosted-snuba-consumer-1 | During handling of the above exception, another exception occurred:
sentry-self-hosted-snuba-consumer-1 |
sentry-self-hosted-snuba-consumer-1 | Traceback (most recent call last):
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 319, in run
sentry-self-hosted-snuba-consumer-1 | self._run_once()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 393, in _run_once
sentry-self-hosted-snuba-consumer-1 | self.__processing_strategy.poll()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/strategies/guard.py", line 101, in poll
sentry-self-hosted-snuba-consumer-1 | self.__inner_strategy.poll()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/strategies/run_task.py", line 55, in poll
sentry-self-hosted-snuba-consumer-1 | self.__next_step.poll()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/strategies/guard.py", line 37, in poll
sentry-self-hosted-snuba-consumer-1 | self.__next_step.poll()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/strategies/reduce.py", line 168, in poll
sentry-self-hosted-snuba-consumer-1 | self.__next_step.poll()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/strategies/run_task_in_threads.py", line 107, in poll
sentry-self-hosted-snuba-consumer-1 | result = future.result()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 437, in result
sentry-self-hosted-snuba-consumer-1 | return self.__get_result()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
sentry-self-hosted-snuba-consumer-1 | raise self._exception
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
sentry-self-hosted-snuba-consumer-1 | result = self.fn(*self.args, **self.kwargs)
sentry-self-hosted-snuba-consumer-1 | File "/usr/src/snuba/snuba/consumers/strategy_factory.py", line 137, in flush_batch
sentry-self-hosted-snuba-consumer-1 | message.payload.close()
sentry-self-hosted-snuba-consumer-1 | File "/usr/src/snuba/snuba/consumers/consumer.py", line 330, in close
sentry-self-hosted-snuba-consumer-1 | self.__insert_batch_writer.close()
sentry-self-hosted-snuba-consumer-1 | File "/usr/src/snuba/snuba/consumers/consumer.py", line 166, in close
sentry-self-hosted-snuba-consumer-1 | self.__writer.write(
sentry-self-hosted-snuba-consumer-1 | File "/usr/src/snuba/snuba/clickhouse/http.py", line 355, in write
sentry-self-hosted-snuba-consumer-1 | batch.join(timeout=batch_join_timeout)
sentry-self-hosted-snuba-consumer-1 | File "/usr/src/snuba/snuba/clickhouse/http.py", line 243, in join
sentry-self-hosted-snuba-consumer-1 | response = self._result.result(timeout)
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 444, in result
sentry-self-hosted-snuba-consumer-1 | return self.__get_result()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
sentry-self-hosted-snuba-consumer-1 | raise self._exception
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
sentry-self-hosted-snuba-consumer-1 | result = self.fn(*self.args, **self.kwargs)
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 787, in urlopen
sentry-self-hosted-snuba-consumer-1 | retries = retries.increment(
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 550, in increment
sentry-self-hosted-snuba-consumer-1 | raise six.reraise(type(error), error, _stacktrace)
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 769, in reraise
sentry-self-hosted-snuba-consumer-1 | raise value.with_traceback(tb)
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
sentry-self-hosted-snuba-consumer-1 | httplib_response = self._make_request(
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request
sentry-self-hosted-snuba-consumer-1 | six.raise_from(e, None)
sentry-self-hosted-snuba-consumer-1 | File "", line 3, in raise_from
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request
sentry-self-hosted-snuba-consumer-1 | httplib_response = conn.getresponse()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/sentry_sdk/integrations/stdlib.py", line 126, in getresponse
sentry-self-hosted-snuba-consumer-1 | rv = real_getresponse(self, *args, **kwargs)
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/http/client.py", line 1348, in getresponse
sentry-self-hosted-snuba-consumer-1 | response.begin()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/http/client.py", line 316, in begin
sentry-self-hosted-snuba-consumer-1 | version, status, reason = self._read_status()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/http/client.py", line 277, in _read_status
sentry-self-hosted-snuba-consumer-1 | line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
sentry-self-hosted-snuba-consumer-1 | return self._sock.recv_into(b)
sentry-self-hosted-snuba-consumer-1 | urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
sentry-self-hosted-snuba-consumer-1 | 2024-04-06 07:01:33,107 Closing <arroyo.backends.kafka.consumer.KafkaConsumer object at 0x7efdad00f280>...
sentry-self-hosted-snuba-consumer-1 | 2024-04-06 07:01:33,108 Partitions to revoke: [Partition(topic=Topic(name='events'), index=0)]
sentry-self-hosted-snuba-consumer-1 | 2024-04-06 07:01:33,108 Partition revocation complete.
sentry-self-hosted-snuba-consumer-1 | 2024-04-06 07:01:33,123 Processor terminated
sentry-self-hosted-snuba-consumer-1 | Traceback (most recent call last):
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
sentry-self-hosted-snuba-consumer-1 | httplib_response = self._make_request(
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request
sentry-self-hosted-snuba-consumer-1 | six.raise_from(e, None)
sentry-self-hosted-snuba-consumer-1 | File "", line 3, in raise_from
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request
sentry-self-hosted-snuba-consumer-1 | httplib_response = conn.getresponse()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/sentry_sdk/integrations/stdlib.py", line 126, in getresponse
sentry-self-hosted-snuba-consumer-1 | rv = real_getresponse(self, *args, **kwargs)
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/http/client.py", line 1348, in getresponse
sentry-self-hosted-snuba-consumer-1 | response.begin()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/http/client.py", line 316, in begin
sentry-self-hosted-snuba-consumer-1 | version, status, reason = self._read_status()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/http/client.py", line 277, in _read_status
sentry-self-hosted-snuba-consumer-1 | line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
sentry-self-hosted-snuba-consumer-1 | return self._sock.recv_into(b)
sentry-self-hosted-snuba-consumer-1 | ConnectionResetError: [Errno 104] Connection reset by peer
sentry-self-hosted-snuba-consumer-1 |
sentry-self-hosted-snuba-consumer-1 | During handling of the above exception, another exception occurred:
sentry-self-hosted-snuba-consumer-1 |
sentry-self-hosted-snuba-consumer-1 | Traceback (most recent call last):
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/bin/snuba", line 33, in
sentry-self-hosted-snuba-consumer-1 | sys.exit(load_entry_point('snuba', 'console_scripts', 'snuba')())
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1130, in call
sentry-self-hosted-snuba-consumer-1 | return self.main(*args, **kwargs)
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1055, in main
sentry-self-hosted-snuba-consumer-1 | rv = self.invoke(ctx)
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
sentry-self-hosted-snuba-consumer-1 | return _process_result(sub_ctx.command.invoke(sub_ctx))
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
sentry-self-hosted-snuba-consumer-1 | return ctx.invoke(self.callback, **ctx.params)
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 760, in invoke
sentry-self-hosted-snuba-consumer-1 | return __callback(*args, **kwargs)
sentry-self-hosted-snuba-consumer-1 | File "/usr/src/snuba/snuba/cli/consumer.py", line 284, in consumer
sentry-self-hosted-snuba-consumer-1 | consumer.run()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 319, in run
sentry-self-hosted-snuba-consumer-1 | self._run_once()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 393, in _run_once
sentry-self-hosted-snuba-consumer-1 | self.__processing_strategy.poll()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/strategies/guard.py", line 101, in poll
sentry-self-hosted-snuba-consumer-1 | self.__inner_strategy.poll()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/strategies/run_task.py", line 55, in poll
sentry-self-hosted-snuba-consumer-1 | self.__next_step.poll()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/strategies/guard.py", line 37, in poll
sentry-self-hosted-snuba-consumer-1 | self.__next_step.poll()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/strategies/reduce.py", line 168, in poll
sentry-self-hosted-snuba-consumer-1 | self.__next_step.poll()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/strategies/run_task_in_threads.py", line 107, in poll
sentry-self-hosted-snuba-consumer-1 | result = future.result()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 437, in result
sentry-self-hosted-snuba-consumer-1 | return self.__get_result()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
sentry-self-hosted-snuba-consumer-1 | raise self._exception
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
sentry-self-hosted-snuba-consumer-1 | result = self.fn(*self.args, **self.kwargs)
sentry-self-hosted-snuba-consumer-1 | File "/usr/src/snuba/snuba/consumers/strategy_factory.py", line 137, in flush_batch
sentry-self-hosted-snuba-consumer-1 | message.payload.close()
sentry-self-hosted-snuba-consumer-1 | File "/usr/src/snuba/snuba/consumers/consumer.py", line 330, in close
sentry-self-hosted-snuba-consumer-1 | self.__insert_batch_writer.close()
sentry-self-hosted-snuba-consumer-1 | File "/usr/src/snuba/snuba/consumers/consumer.py", line 166, in close
sentry-self-hosted-snuba-consumer-1 | self.__writer.write(
sentry-self-hosted-snuba-consumer-1 | File "/usr/src/snuba/snuba/clickhouse/http.py", line 355, in write
sentry-self-hosted-snuba-consumer-1 | batch.join(timeout=batch_join_timeout)
sentry-self-hosted-snuba-consumer-1 | File "/usr/src/snuba/snuba/clickhouse/http.py", line 243, in join
sentry-self-hosted-snuba-consumer-1 | response = self._result.result(timeout)
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 444, in result
sentry-self-hosted-snuba-consumer-1 | return self.__get_result()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
sentry-self-hosted-snuba-consumer-1 | raise self._exception
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
sentry-self-hosted-snuba-consumer-1 | result = self.fn(*self.args, **self.kwargs)
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 787, in urlopen
sentry-self-hosted-snuba-consumer-1 | retries = retries.increment(
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 550, in increment
sentry-self-hosted-snuba-consumer-1 | raise six.reraise(type(error), error, _stacktrace)
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 769, in reraise
sentry-self-hosted-snuba-consumer-1 | raise value.with_traceback(tb)
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
sentry-self-hosted-snuba-consumer-1 | httplib_response = self._make_request(
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request
sentry-self-hosted-snuba-consumer-1 | six.raise_from(e, None)
sentry-self-hosted-snuba-consumer-1 | File "", line 3, in raise_from
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request
sentry-self-hosted-snuba-consumer-1 | httplib_response = conn.getresponse()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/site-packages/sentry_sdk/integrations/stdlib.py", line 126, in getresponse
sentry-self-hosted-snuba-consumer-1 | rv = real_getresponse(self, *args, **kwargs)
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/http/client.py", line 1348, in getresponse
sentry-self-hosted-snuba-consumer-1 | response.begin()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/http/client.py", line 316, in begin
sentry-self-hosted-snuba-consumer-1 | version, status, reason = self._read_status()
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/http/client.py", line 277, in _read_status
sentry-self-hosted-snuba-consumer-1 | line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
sentry-self-hosted-snuba-consumer-1 | File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
sentry-self-hosted-snuba-consumer-1 | return self._sock.recv_into(b)
sentry-self-hosted-snuba-consumer-1 | urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
sentry-self-hosted-snuba-consumer-1 | 2024-04-06 07:01:38,765 Initializing Snuba...
sentry-self-hosted-snuba-consumer-1 | 2024-04-06 07:01:42,706 Snuba initialization took 3.9428554102778435s
sentry-self-hosted-snuba-consumer-1 | 2024-04-06 07:01:43,217 Initializing Snuba...
sentry-self-hosted-snuba-consumer-1 | 2024-04-06 07:01:46,620 Snuba initialization took 3.4044318441301584s
sentry-self-hosted-snuba-consumer-1 | 2024-04-06 07:01:46,629 Consumer Starting
sentry-self-hosted-snuba-consumer-1 | 2024-04-06 07:01:46,629 Checking Clickhouse connections...
sentry-self-hosted-snuba-consumer-1 | 2024-04-06 07:01:46,637 Successfully connected to Clickhouse: cluster_name=None
sentry-self-hosted-snuba-consumer-1 | 2024-04-06 07:01:46,638 librdkafka log level: 6
sentry-self-hosted-snuba-consumer-1 | 2024-04-06 07:01:46,762 New partitions assigned: {Partition(topic=Topic(name='events'), index=0): 13754531}
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/http/client.py", line 316, in begin
sentry-self-hosted-snuba-outcomes-consumer-1 | version, status, reason = self._read_status()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/http/client.py", line 277, in _read_status
sentry-self-hosted-snuba-outcomes-consumer-1 | line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
sentry-self-hosted-snuba-outcomes-consumer-1 | return self._sock.recv_into(b)
sentry-self-hosted-snuba-outcomes-consumer-1 | ConnectionResetError: [Errno 104] Connection reset by peer
sentry-self-hosted-snuba-outcomes-consumer-1 |
sentry-self-hosted-snuba-outcomes-consumer-1 | During handling of the above exception, another exception occurred:
sentry-self-hosted-snuba-outcomes-consumer-1 |
sentry-self-hosted-snuba-outcomes-consumer-1 | Traceback (most recent call last):
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/bin/snuba", line 33, in
sentry-self-hosted-snuba-outcomes-consumer-1 | sys.exit(load_entry_point('snuba', 'console_scripts', 'snuba')())
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1130, in call
sentry-self-hosted-snuba-outcomes-consumer-1 | return self.main(*args, **kwargs)
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1055, in main
sentry-self-hosted-snuba-outcomes-consumer-1 | rv = self.invoke(ctx)
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
sentry-self-hosted-snuba-outcomes-consumer-1 | return _process_result(sub_ctx.command.invoke(sub_ctx))
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
sentry-self-hosted-snuba-outcomes-consumer-1 | return ctx.invoke(self.callback, **ctx.params)
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/click/core.py", line 760, in invoke
sentry-self-hosted-snuba-outcomes-consumer-1 | return __callback(*args, **kwargs)
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/src/snuba/snuba/cli/consumer.py", line 284, in consumer
sentry-self-hosted-snuba-outcomes-consumer-1 | consumer.run()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 319, in run
sentry-self-hosted-snuba-outcomes-consumer-1 | self._run_once()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/processor.py", line 393, in _run_once
sentry-self-hosted-snuba-outcomes-consumer-1 | self.__processing_strategy.poll()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/strategies/guard.py", line 101, in poll
sentry-self-hosted-snuba-outcomes-consumer-1 | self.__inner_strategy.poll()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/strategies/run_task.py", line 55, in poll
sentry-self-hosted-snuba-outcomes-consumer-1 | self.__next_step.poll()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/strategies/guard.py", line 37, in poll
sentry-self-hosted-snuba-outcomes-consumer-1 | self.__next_step.poll()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/strategies/reduce.py", line 168, in poll
sentry-self-hosted-snuba-outcomes-consumer-1 | self.__next_step.poll()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/arroyo/processing/strategies/run_task_in_threads.py", line 107, in poll
sentry-self-hosted-snuba-outcomes-consumer-1 | result = future.result()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 437, in result
sentry-self-hosted-snuba-outcomes-consumer-1 | return self.__get_result()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
sentry-self-hosted-snuba-outcomes-consumer-1 | raise self._exception
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
sentry-self-hosted-snuba-outcomes-consumer-1 | result = self.fn(*self.args, **self.kwargs)
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/src/snuba/snuba/consumers/strategy_factory.py", line 137, in flush_batch
sentry-self-hosted-snuba-outcomes-consumer-1 | message.payload.close()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/src/snuba/snuba/consumers/consumer.py", line 330, in close
sentry-self-hosted-snuba-outcomes-consumer-1 | self.__insert_batch_writer.close()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/src/snuba/snuba/consumers/consumer.py", line 166, in close
sentry-self-hosted-snuba-outcomes-consumer-1 | self.__writer.write(
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/src/snuba/snuba/clickhouse/http.py", line 355, in write
sentry-self-hosted-snuba-outcomes-consumer-1 | batch.join(timeout=batch_join_timeout)
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/src/snuba/snuba/clickhouse/http.py", line 243, in join
sentry-self-hosted-snuba-outcomes-consumer-1 | response = self._result.result(timeout)
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 444, in result
sentry-self-hosted-snuba-outcomes-consumer-1 | return self.__get_result()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
sentry-self-hosted-snuba-outcomes-consumer-1 | raise self._exception
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
sentry-self-hosted-snuba-outcomes-consumer-1 | result = self.fn(*self.args, **self.kwargs)
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 787, in urlopen
sentry-self-hosted-snuba-outcomes-consumer-1 | retries = retries.increment(
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 550, in increment
sentry-self-hosted-snuba-outcomes-consumer-1 | raise six.reraise(type(error), error, _stacktrace)
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 769, in reraise
sentry-self-hosted-snuba-outcomes-consumer-1 | raise value.with_traceback(tb)
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
sentry-self-hosted-snuba-outcomes-consumer-1 | httplib_response = self._make_request(
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 449, in _make_request
sentry-self-hosted-snuba-outcomes-consumer-1 | six.raise_from(e, None)
sentry-self-hosted-snuba-outcomes-consumer-1 | File "", line 3, in raise_from
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 444, in _make_request
sentry-self-hosted-snuba-outcomes-consumer-1 | httplib_response = conn.getresponse()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/site-packages/sentry_sdk/integrations/stdlib.py", line 126, in getresponse
sentry-self-hosted-snuba-outcomes-consumer-1 | rv = real_getresponse(self, *args, **kwargs)
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/http/client.py", line 1348, in getresponse
sentry-self-hosted-snuba-outcomes-consumer-1 | response.begin()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/http/client.py", line 316, in begin
sentry-self-hosted-snuba-outcomes-consumer-1 | version, status, reason = self._read_status()
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/http/client.py", line 277, in _read_status
sentry-self-hosted-snuba-outcomes-consumer-1 | line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
sentry-self-hosted-snuba-outcomes-consumer-1 | File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
sentry-self-hosted-snuba-outcomes-consumer-1 | return self._sock.recv_into(b)
sentry-self-hosted-snuba-outcomes-consumer-1 | urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
sentry-self-hosted-snuba-outcomes-consumer-1 | 2024-04-06 07:01:38,765 Initializing Snuba...
sentry-self-hosted-snuba-outcomes-consumer-1 | 2024-04-06 07:01:42,677 Snuba initialization took 3.913459099829197s
sentry-self-hosted-snuba-outcomes-consumer-1 | 2024-04-06 07:01:43,191 Initializing Snuba...
sentry-self-hosted-snuba-outcomes-consumer-1 | 2024-04-06 07:01:46,586 Snuba initialization took 3.3963669184595346s
sentry-self-hosted-snuba-outcomes-consumer-1 | 2024-04-06 07:01:46,595 Consumer Starting
sentry-self-hosted-snuba-outcomes-consumer-1 | 2024-04-06 07:01:46,595 Checking Clickhouse connections...
sentry-self-hosted-snuba-outcomes-consumer-1 | 2024-04-06 07:01:46,605 Successfully connected to Clickhouse: cluster_name=None
sentry-self-hosted-snuba-outcomes-consumer-1 | 2024-04-06 07:01:46,605 librdkafka log level: 6
sentry-self-hosted-snuba-outcomes-consumer-1 | 2024-04-06 07:01:46,762 New partitions assigned: {Partition(topic=Topic(name='outcomes'), index=0): 27696545}
sentry-self-hosted-snuba-replays-consumer-1 | 2024-04-06 07:01:34,521 Partitions to revoke: [Partition(topic=Topic(name='ingest-replay-events'), index=0)]
sentry-self-hosted-snuba-replays-consumer-1 | 2024-04-06 07:01:34,522 Closing <arroyo.processing.strategies.guard.StrategyGuard object at 0x7f28fc79f190>...
sentry-self-hosted-snuba-replays-consumer-1 | 2024-04-06 07:01:34,522 Waiting for <arroyo.processing.strategies.guard.StrategyGuard object at 0x7f28fc79f190> to exit...
sentry-self-hosted-snuba-replays-consumer-1 | 2024-04-06 07:01:34,523 <arroyo.processing.strategies.guard.StrategyGuard object at 0x7f28fc79f190> exited successfully, releasing assignment.
sentry-self-hosted-snuba-replays-consumer-1 | 2024-04-06 07:01:34,523 Partition revocation complete.
sentry-self-hosted-snuba-replays-consumer-1 | 2024-04-06 07:01:34,758 New partitions assigned: {Partition(topic=Topic(name='ingest-replay-events'), index=0): 552065}
sentry-self-hosted-snuba-replays-consumer-1 | 2024-04-06 07:01:46,757 Partitions to revoke: [Partition(topic=Topic(name='ingest-replay-events'), index=0)]
sentry-self-hosted-snuba-replays-consumer-1 | 2024-04-06 07:01:46,758 Closing <arroyo.processing.strategies.guard.StrategyGuard object at 0x7f28fc79f700>...
sentry-self-hosted-snuba-replays-consumer-1 | 2024-04-06 07:01:46,758 Waiting for <arroyo.processing.strategies.guard.StrategyGuard object at 0x7f28fc79f700> to exit...
sentry-self-hosted-snuba-replays-consumer-1 | 2024-04-06 07:01:46,758 <arroyo.processing.strategies.guard.StrategyGuard object at 0x7f28fc79f700> exited successfully, releasing assignment.
sentry-self-hosted-snuba-replays-consumer-1 | 2024-04-06 07:01:46,758 Partition revocation complete.
sentry-self-hosted-snuba-replays-consumer-1 | 2024-04-06 07:01:46,762 New partitions assigned: {Partition(topic=Topic(name='ingest-replay-events'), index=0): 552065}
sentry-self-hosted-events-consumer-1 | 22:40:54 [WARNING] sentry.ingest.consumer.processors: pre-process-forwarder detected a duplicated event with id:25eac491a7a44dfd9facd3591cb9e30d for project:13.
sentry-self-hosted-events-consumer-1 | 02:47:40 [WARNING] sentry.ingest.consumer.processors: pre-process-forwarder detected a duplicated event with id:ee7d095b4cc64aa3ada5ab155e38e023 for project:10.
sentry-self-hosted-events-consumer-1 | 04:01:58 [WARNING] sentry.ingest.consumer.processors: pre-process-forwarder detected a duplicated event with id:c21a9456665d4024bb413bf823b7441d for project:14.
sentry-self-hosted-events-consumer-1 | 04:36:06 [WARNING] sentry.ingest.consumer.processors: pre-process-forwarder detected a duplicated event with id:3bd64b2e09f846c19d29a56122e53886 for project:10.
sentry-self-hosted-snuba-profiling-functions-consumer-1 | 2024-04-06 07:01:34,750 Partitions to revoke: [Partition(topic=Topic(name='profiles-call-tree'), index=0)]
sentry-self-hosted-snuba-profiling-functions-consumer-1 | 2024-04-06 07:01:34,751 Closing <arroyo.processing.strategies.guard.StrategyGuard object at 0x7fc94a9e3370>...
sentry-self-hosted-snuba-profiling-functions-consumer-1 | 2024-04-06 07:01:34,751 Waiting for <arroyo.processing.strategies.guard.StrategyGuard object at 0x7fc94a9e3370> to exit...
sentry-self-hosted-snuba-profiling-functions-consumer-1 | 2024-04-06 07:01:34,751 <arroyo.processing.strategies.guard.StrategyGuard object at 0x7fc94a9e3370> exited successfully, releasing assignment.
sentry-self-hosted-snuba-profiling-functions-consumer-1 | 2024-04-06 07:01:34,751 Partition revocation complete.
sentry-self-hosted-snuba-profiling-functions-consumer-1 | 2024-04-06 07:01:35,246 New partitions assigned: {Partition(topic=Topic(name='profiles-call-tree'), index=0): 0}
sentry-self-hosted-snuba-profiling-functions-consumer-1 | 2024-04-06 07:01:46,755 Partitions to revoke: [Partition(topic=Topic(name='profiles-call-tree'), index=0)]
sentry-self-hosted-snuba-profiling-functions-consumer-1 | 2024-04-06 07:01:46,756 Closing <arroyo.processing.strategies.guard.StrategyGuard object at 0x7fc94a9e35e0>...
sentry-self-hosted-snuba-profiling-functions-consumer-1 | 2024-04-06 07:01:46,756 Waiting for <arroyo.processing.strategies.guard.StrategyGuard object at 0x7fc94a9e35e0> to exit...
sentry-self-hosted-snuba-profiling-functions-consumer-1 | 2024-04-06 07:01:46,756 <arroyo.processing.strategies.guard.StrategyGuard object at 0x7fc94a9e35e0> exited successfully, releasing assignment.
sentry-self-hosted-snuba-profiling-functions-consumer-1 | 2024-04-06 07:01:46,756 Partition revocation complete.
sentry-self-hosted-snuba-profiling-functions-consumer-1 | 2024-04-06 07:01:46,764 New partitions assigned: {Partition(topic=Topic(name='profiles-call-tree'), index=0): 0}
sentry-self-hosted-symbolicator-cleanup-1 | 2024-04-05T23:55:02.023164741Z INFO symbolicator_service::caching::cleanup: Cleaning up cficaches cache
sentry-self-hosted-symbolicator-cleanup-1 | 2024-04-05T23:55:02.023849214Z INFO symbolicator_service::caching::cleanup: Cleaning up cficaches complete
sentry-self-hosted-symbolicator-cleanup-1 | 2024-04-05T23:55:02.023867955Z INFO symbolicator_service::caching::cleanup: Retained 0 directories and 0 files, totaling 0 bytes
sentry-self-hosted-symbolicator-cleanup-1 | 2024-04-05T23:55:02.023880595Z INFO symbolicator_service::caching::cleanup: Removed 0 directories and 0 files, totaling 0 bytes
sentry-self-hosted-symbolicator-cleanup-1 | 2024-04-05T23:55:02.023886309Z INFO symbolicator_service::caching::cleanup: Cleaning up auxdifs cache
sentry-self-hosted-symbolicator-cleanup-1 | 2024-04-05T23:55:02.024228737Z INFO symbolicator_service::caching::cleanup: Cleaning up auxdifs complete
sentry-self-hosted-symbolicator-cleanup-1 | 2024-04-05T23:55:02.02423951Z INFO symbolicator_service::caching::cleanup: Retained 0 directories and 0 files, totaling 0 bytes
sentry-self-hosted-symbolicator-cleanup-1 | 2024-04-05T23:55:02.024243143Z INFO symbolicator_service::caching::cleanup: Removed 0 directories and 0 files, totaling 0 bytes
sentry-self-hosted-symbolicator-cleanup-1 | 2024-04-05T23:55:02.024246999Z INFO symbolicator_service::caching::cleanup: Cleaning up il2cpp cache
sentry-self-hosted-symbolicator-cleanup-1 | 2024-04-05T23:55:02.024740006Z INFO symbolicator_service::caching::cleanup: Cleaning up il2cpp complete
sentry-self-hosted-symbolicator-cleanup-1 | 2024-04-05T23:55:02.024750734Z INFO symbolicator_service::caching::cleanup: Retained 0 directories and 0 files, totaling 0 bytes
sentry-self-hosted-symbolicator-cleanup-1 | 2024-04-05T23:55:02.024754166Z INFO symbolicator_service::caching::cleanup: Removed 0 directories and 0 files, totaling 0 bytes
sentry-self-hosted-symbolicator-cleanup-1 | 2024-04-05T23:55:02.024757338Z INFO symbolicator_service::caching::cleanup: Cleaning up diagnostics cache
sentry-self-hosted-symbolicator-cleanup-1 | 2024-04-05T23:55:02.025249628Z INFO symbolicator_service::caching::cleanup: Cleaning up diagnostics complete

Event ID

No response

[azaslavsky]

Can you check your memory usage? Often this happens when the system or container is OOM.