New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] 多并发调用,偶尔出现因为embedding没加载而调用失败 #3899
Comments
The error you're encountering, "AttributeError: 'NoneType' object has no attribute 'acquire'", indicates that the
By following these steps, you should be able to mitigate the intermittent failures related to embeddings not being loaded properly when handling multiple concurrent requests with vllm acceleration.
|
这个框架没做并发啊 |
问题描述 / Problem Description
开启vllm加速启动服务后:
多并发调用,则会因为embedding没加载而调用失败
复现问题的步骤 / Steps to Reproduce
1.设置vllm加速:
FSCHAT_MODEL_WORKERS = {
"default": {
"host": DEFAULT_BIND_HOST,
"port": 30002,
"device": LLM_DEVICE,
"infer_turbo": 'vllm',
},
2.启动服务:
python startup.py -a
3.python代码多并发调用
预期的结果 / Expected Result
正常返回生成的答案
实际结果 / Actual Result
多并发请求服务,偶尔能正常执行完,偶尔会部分正常执行,部分报错:
AttributeError: 'NoneType' object has no attribute 'acquire'
具体报错信息:
2024-04-26 07:13:11,178 - _client.py[line:1758] - INFO: HTTP Request: POST http://127.0.0.1:30000/v1/chat/completions "HTTP/1.1 200 OK"
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/sse_starlette/sse.py", line 269, in call
await wrap(partial(self.listen_for_disconnect, receive))
File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/sse_starlette/sse.py", line 258, in wrap
await func()
File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/sse_starlette/sse.py", line 215, in listen_for_disconnect
message = await receive()
^^^^^^^^^^^^^^^
File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 568, in receive
await self.message_event.wait()
File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/asyncio/locks.py", line 213, in wait
await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 7f1478689710
During handling of the above exception, another exception occurred:
| File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 411, in run_asgi
| result = await app( # type: ignore[func-returns-value]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 69, in call
| return await self.app(scope, receive, send)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in call
| await super().call(scope, receive, send)
| File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/starlette/applications.py", line 119, in call
| await self.middleware_stack(scope, receive, send)
| File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in call
| raise exc
| File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in call
| await self.app(scope, receive, _send)
| File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in call
| await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
| File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
| raise exc
| File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| await app(scope, receive, sender)
| File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/starlette/routing.py", line 762, in call
| await self.middleware_stack(scope, receive, send)
| File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/starlette/routing.py", line 782, in app
| await route.handle(scope, receive, send)
| File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
| await self.app(scope, receive, send)
| File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
| await wrap_app_handling_exceptions(app, request)(scope, receive, send)
| File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
| raise exc
| File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| await app(scope, receive, sender)
| File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/starlette/routing.py", line 75, in app
| await response(scope, receive, send)
| File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/sse_starlette/sse.py", line 255, in call
| async with anyio.create_task_group() as task_group:
| File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 678, in aexit
| raise BaseExceptionGroup(
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/sse_starlette/sse.py", line 258, in wrap
| await func()
| File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/sse_starlette/sse.py", line 245, in stream_response
| async for data in self.body_iterator:
| File "/home/algo/workproject/Langchain-Chatchat-gen-v1.0.1/server/chat/knowledge_base_chat.py", line 109, in knowledge_base_chat_iterator
| docs = await run_in_threadpool(search_docs,
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/starlette/concurrency.py", line 40, in run_in_threadpool
| return await anyio.to_thread.run_sync(func, *args)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
| return await get_async_backend().run_sync_in_worker_thread(
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2134, in run_sync_in_worker_thread
| return await future
| ^^^^^^^^^^^^
| File "/home/algo/anaconda3/envs/chatfaqgpu/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 851, in run
| result = context.run(func, *args)
| ^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/algo/workproject/Langchain-Chatchat-gen-v1.0.1/server/knowledge_base/kb_doc_api.py", line 38, in search_docs
| docs = kb.search_docs(query, top_k, score_threshold)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/algo/workproject/Langchain-Chatchat-gen-v1.0.1/server/knowledge_base/kb_service/base.py", line 182, in search_docs
| docs = self.do_search(query, top_k, score_threshold)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/algo/workproject/Langchain-Chatchat-gen-v1.0.1/server/knowledge_base/kb_service/faiss_kb_service.py", line 67, in do_search
| with self.load_vector_store().acquire() as vs:
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| AttributeError: 'NoneType' object has no attribute 'acquire'
环境信息 / Environment Information
附加信息 / Additional Information
添加与问题相关的任何其他信息 / Add any other information related to the issue.
The text was updated successfully, but these errors were encountered: