Are async path operations supposed to be exclusively single-threaded?

[Minibrams]

First Check

• I added a very descriptive title to this issue.
• I used the GitHub search to find a similar issue and didn't find it.
• I searched the FastAPI documentation, with the integrated search.
• I already searched in Google "How to X in FastAPI" and didn't find any information.
• I already read and followed all the tutorial in the docs and didn't find an answer.
• I already checked if it is not related to FastAPI but to Pydantic.
• I already checked if it is not related to FastAPI but to Swagger UI.
• I already checked if it is not related to FastAPI but to ReDoc.

Commit to Help

• I commit to help with one of those options 👆

Example Code

# Will run multithreaded, low response times
@app.get('/sync/http')
def http_sync():
    with Client() as http:
        return http.get('http://some-host.com').content

# Will run single-threaded, long response times under heavy load
@app.get('/async/http')
async def http_async():
    async with AsyncClient() as http:
        return (await http.get('http://some-host.com')).content

# Will block the entire server, requests will fail and response times approach minutes under heavy load
@app.get('/async/http/sync')
async def http_async_sync():
    with Client() as http:
        return http.get('http://some-host.com').content
 

Description

I've been using FastAPI for a bit now, and it's great. However, I've always been a little confused about the proper mental model regarding async def and def endpoints. With issues like #2619 and #3725, I don't seem to be alone in this.

With all the magic that FastAPI is doing behind the scenes - and given the "you can mix def and async def in your path operations ... FastAPI will know what to do with them" from the documentation, one can easily assume that whatever they do, it will work and run as fast as advertised.

However, after load testing some different implementations, reality seems to be a little more confusing (to me).

I decided to test different combinations of async/sync path operations and IO to get a better idea of how things work.
The tests are available here along with a comprehensive description of setup and results if you want to review the code and run it locally.

TL;DR: The tests simulate 500 users that each hit different combinations of def/async def path operations with def/async def IO being done inside them. Tests run for 1 minute.
The IO in question is an HTTP GET request to an external service that sleeps for 1 second on requests before returning. Here are the results:

	sync endpoint w. sync io	async endpoint w. async io	async endpoint w. sync io
Num. users	500	500	500
Test runtime	1 minute	1 minute	1 minute
Num. requests	2122	3190	785
Num. failures	0 (0%)	0 (0%)	732 (93.2%)
Avg. num. requests/sec	35.6	53.2	13.1
Avg. num. failures/sec	0	0	12.2
Median response time	1300 ms	8300 ms	27000 ms
Num. threads spawned	41	1	1
Avg. num. active threads	41	1	1

We can make the following observations:

1. def endpoints are run in a threadpool, as documented, allowing us to do blocking IO without it blocking the server
2. async def endpoints run in a single thread, each request seemingly a co-routine
   • This also means that a single blocking IO operation in an async def endpoint will block the entire server

Here's my primary confusion: I understand that async really only makes sense in the context of a single thread. However, only allocating one single thread for every single request that hits an async def endpoint seems strange to me, and leads to very long response times if requests start piling up. The way I see it, the only way I would be able to scale my async def endpoint would be to allocate more workers with a PM like Gunicorn, but spinning up a new worker seems like incredible overkill when I could alternatively "just" make a synchronous def end let the request be handled in its own thread.

In most of the documentation (not just FastAPI's documentation), async/await is recommended for IO work, which makes sense since we're doing a lot of waiting anyway. However, I think that point falls to the ground when our only option is to do async/await within a single thread when we seem to have the ability to spawn threads at our leisure with synchronous def endpoints, yielding much lower response times even though all the work being done is IO.

Save for the case where we would like to make X HTTP requests or launch X database queries at the same time and gather asynchronously, async/await makes little sense if every request is handled in its own thread. But I also think that async/await makes very little practical sense if every request can only be handled by the same single thread without any option to increase capacity.

Is my understanding correct, or am I supposed to handle asynchronous work in a different way?

If I understand the situation correctly, maybe a hybrid could be considered, e.g. allocating a smaller threadpool for async def path operations, allowing requests to be handled asynchronously within the same thread, but also allowing the number of threads to scale?

Operating System

macOS

Operating System Details

No response

FastAPI Version

0.70.0

Python Version

Python 3.8.5

Additional Context

No response

[raphaelauv]

it's easy

python have a GIL , that mean multithreading is "fake"
AND
async give us non active waiting for IO operations
SO
having 1 async loop per python process is the best you can do
SO
if you also have non-async work, this must not run with the same thread than the main async loop ( so it do not completely block it )
SO
every none async endpoint is run in a thread-pool provided by starlette (anyio)
BUT
If you really have heavy IO non async operations you should probably not run it in the same process of the main async loop
THEN
look to this encode/starlette#1094

By default you should run 1 fastAPI process per core , so 1 async loop per CPU core, for that use uvicorn or gunicorn as you want.
BUT
if you deploy with K8S only run 1 uvicorn process with only ONE worker and scale the number of pods EXCEPT if you have heavy memory usage then sharing memory with multiple worker in the same container is probably better.

CONCLUSION :

trade off !

fastAPI by default is doing what we expect of a web framework , YOU have to custom it if necessary.

Thank you. bye

[insomnes]

https://blog.miguelgrinberg.com/post/ignore-all-web-performance-benchmarks-including-this-one

[wonrax]

Nice and detailed write up, I wonder how things are now. I feel like it's almost impossible to find documents for stuff like this, let alone API for configurations. For instance, in Rust, specifically the Tokio runtime, there are options to change the size of both the asynchronous threadpool and the blocking threadpool.