Armeria - blockling task executor

[Sethuraman]

Hi

I am trying to get some advice on event loop threads and blocking task executors. The general advice for armeria based on docs and prior discussions is not to block the event loop threads.

We currently have a service that uses a blocking task executor for all exposed endpoints. Some of these calls are processing events from kafka and writing them down in a db. Some other calls are grpc endpoints that retrieve data from the DB.

We had an issue where the event writes were taking too much DB time and timing out (lock time outs). This resulted in the grpc read calls to respond back with RST_STREAM errors. Basically the clients making the grpc calls were timing out.

We were thinking, the event processor threads were not allowing the reads threads to proceed.

The event processor threads and the read threads are blocking event loop threads. If we continue to let them be blocking but whenever a DB call is being made from these threads we offload the DB call onto another thread pool, would that help? Or would we have to stop using blocking executors all together.

What would that actually mean? Would it change the way we implement event processing + read api calls altogether?

[trustin]

Hi! Thank for asking a great question, @Sethuraman. If I understood correctly, you have two types of blocking tasks here:

• Consume Kafka queue to write into DB
• Read from DB

and if one type of tasks (A) dominates the blocking task executor, the other type of tasks (B) will not have a chance to be executed, because the executor is busy handling (A).

In such as case, it is recommended to use a different thread pool for different type of tasks. For example, you could create another thread pool to handle the task type (B).

[Sethuraman]

That makes sense. 2 follow up questions -

1. Is there some example code that demos how we can use 2 different thread pools for this?

2. If a thread, let's say the kafka thread, picks up an event to process, it then runs up to the point where it makes a DB call. At that point, it moves execution of the DB call to another thread pool. We use Kotlin - so the Dispatcher.IO thread pool. When this thread switch happens, does it free the original event loop thread to process something else? Or will that thread wait (block) for the DB thread to complete before returning back with the response?

[trustin]

Oops, sorry for a late reply, @Sethuraman. 🙇

Is there some example code that demos how we can use 2 different thread pools for this?

We don't have one, but it is simply having two thread pools (ExecutorServices) and then submitting (.execute(task)) the tasks to one of them depending on the task type.

does it free the original event loop thread to process something else?

I guess it will free the event loop but I'd recommend you to check if it really does so since it's implementation detail. Perhaps you could use Blockhound to see if anything blocks the event loop?