Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Concurrent testing,asyncio.exceptions.CancelledError: Cancelled by cancel scope 10abec670 #80

Open
YangZhiBoGreenHand opened this issue Apr 1, 2024 · 2 comments
Assignees
Labels
bug Something isn't working

Comments

@YangZhiBoGreenHand
Copy link

YangZhiBoGreenHand commented Apr 1, 2024

Describe the bug
asyncio.exceptions.CancelledError

To Reproduce
Steps to reproduce the behavior:

  1. Concurrent test an assistant, create chat concurrently, create messages, and generate results
  2. After the end, the server will report an error

Expected behavior
A clear and concise description of what you expected to happen.
INFO: 127.0.0.1:51131 - "POST /api/v1/assistants/X5lMSQinHLE8beIWz7mjDqxM/chats HTTP/1.1" 200 OK
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/aioredis/connection.py", line 860, in send_packed_command
await asyncio.wait_for(
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/asyncio/tasks.py", line 442, in wait_for
return await fut
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/aioredis/connection.py", line 842, in _send_packed_command
await self._writer.drain()
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/asyncio/streams.py", line 387, in drain
await self._protocol._drain_helper()
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/asyncio/streams.py", line 190, in _drain_helper
raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in call
return await self.app(scope, receive, send)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/starlette/applications.py", line 123, in call
await self.middleware_stack(scope, receive, send)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/starlette/middleware/errors.py", line 164, in call
await self.app(scope, receive, _send)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/starlette/middleware/cors.py", line 83, in call
await self.app(scope, receive, send)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 62, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/starlette/routing.py", line 758, in call
await self.middleware_stack(scope, receive, send)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/starlette/routing.py", line 778, in app
await route.handle(scope, receive, send)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/starlette/routing.py", line 299, in handle
await self.app(scope, receive, send)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/starlette/routing.py", line 79, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/starlette/routing.py", line 74, in app
response = await func(request)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/fastapi/routing.py", line 294, in app
raw_response = await run_endpoint_function(
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
return await dependant.call(**values)
File "/Users/yangzhibo/Projects/TaskingAI/backend/app/routes/auto/message.py", line 114, in api_create
entity = await ops.create(
File "/Users/yangzhibo/Projects/TaskingAI/backend/app/operators/assistant/message.py", line 35, in create
chat = await chat_ops.get(
File "/Users/yangzhibo/Projects/TaskingAI/backend/tkhelper/models/operator/postgres_operator.py", line 102, in get
entity = await self.redis.get(**kwargs)
File "/Users/yangzhibo/Projects/TaskingAI/backend/tkhelper/models/redis.py", line 49, in get
data = await self.redis_conn.get_object(key)
File "/Users/yangzhibo/Projects/TaskingAI/backend/tkhelper/database/redis/connection.py", line 94, in get_object
value_string = await self.redis.get(key)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/aioredis/client.py", line 1084, in execute_command
await conn.send_command(*args)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/aioredis/connection.py", line 885, in send_command
await self.send_packed_command(
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/aioredis/connection.py", line 868, in send_packed_command
await self.disconnect()
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/aioredis/connection.py", line 806, in disconnect
await self._writer.wait_closed() # type: ignore[union-attr]
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/asyncio/streams.py", line 359, in wait_closed
await self._protocol._get_close_waiter(self)
asyncio.exceptions.CancelledError: Cancelled by cancel scope 10abec670
2024-04-01 12:40:37,629 - asyncio - WARNING - socket.send() raised exception.
2024-04-01 12:40:37,646 - asyncio - WARNING - socket.send() raised exception.
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/aioredis/connection.py", line 860, in send_packed_command
await asyncio.wait_for(
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/asyncio/tasks.py", line 442, in wait_for
return await fut
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/aioredis/connection.py", line 842, in _send_packed_command
await self._writer.drain()
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/asyncio/streams.py", line 387, in drain
await self._protocol._drain_helper()
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/asyncio/streams.py", line 190, in _drain_helper
raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in call
return await self.app(scope, receive, send)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/starlette/applications.py", line 123, in call
await self.middleware_stack(scope, receive, send)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/starlette/middleware/errors.py", line 164, in call
await self.app(scope, receive, _send)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/starlette/middleware/cors.py", line 83, in call
await self.app(scope, receive, send)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 62, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/starlette/routing.py", line 758, in call
await self.middleware_stack(scope, receive, send)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/starlette/routing.py", line 778, in app
await route.handle(scope, receive, send)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/starlette/routing.py", line 299, in handle
await self.app(scope, receive, send)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/starlette/routing.py", line 79, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/starlette/routing.py", line 74, in app
response = await func(request)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/fastapi/routing.py", line 294, in app
raw_response = await run_endpoint_function(
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
return await dependant.call(**values)
File "/Users/yangzhibo/Projects/TaskingAI/backend/app/routes/auto/message.py", line 114, in api_create
entity = await ops.create(
File "/Users/yangzhibo/Projects/TaskingAI/backend/app/operators/assistant/message.py", line 60, in create
await update_chat_memory(chat=chat, memory=updated_chat_memory)
File "/Users/yangzhibo/Projects/TaskingAI/backend/app/services/assistant/chat.py", line 22, in update_chat_memory
chat = await chat_ops.update(
File "/Users/yangzhibo/Projects/TaskingAI/backend/tkhelper/models/operator/postgres_operator.py", line 53, in update
entity = await self.get(**kwargs, postgres_conn=conn)
File "/Users/yangzhibo/Projects/TaskingAI/backend/tkhelper/models/operator/postgres_operator.py", line 102, in get
entity = await self.redis.get(**kwargs)
File "/Users/yangzhibo/Projects/TaskingAI/backend/tkhelper/models/redis.py", line 49, in get
data = await self.redis_conn.get_object(key)
File "/Users/yangzhibo/Projects/TaskingAI/backend/tkhelper/database/redis/connection.py", line 94, in get_object
value_string = await self.redis.get(key)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/aioredis/client.py", line 1084, in execute_command
await conn.send_command(*args)
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/aioredis/connection.py", line 885, in send_command
await self.send_packed_command(
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/aioredis/connection.py", line 868, in send_packed_command
await self.disconnect()
File "/Users/yangzhibo/Projects/TaskingAI/backend/.venv/lib/python3.9/site-packages/aioredis/connection.py", line 806, in disconnect
await self._writer.wait_closed() # type: ignore[union-attr]
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/asyncio/streams.py", line 359, in wait_closed
await self._protocol._get_close_waiter(self)
asyncio.exceptions.CancelledError: Cancelled by cancel scope 10abec670
2024-04-01 12:40:37,681 - asyncio - WARNING - socket.send() raised exception.

@DynamesC
Copy link
Collaborator

DynamesC commented Apr 1, 2024

Thanks for the feedback yangzhibo. We are aware of this issue, and have already fixed this in an un-released version. That version is estimated to be released in one week, along with several other features and improvements. Please stay tuned.

@jameszyao jameszyao added the bug Something isn't working label Apr 1, 2024
@jameszyao
Copy link
Contributor

@YangZhiBoGreenHand In the v0.2.2 version, we have enhanced the stability of the redis client to avoid continued client downtime. Please check :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants