How does the dispatcher know when to exit?

[wlandau]

While developing crew, sometimes my local R session crashes and the mirai dispatcher keeps running. (But very seldom, and I have not tried since mirai > 0.9.0.9027.) That makes me wonder: what is the mechanism that the dispatcher uses to figure out when to terminate itself?

[wlandau]

If I were implementing this myself, I would try to wake up the dispatcher in response to a pipe event from the client (over the IPC/abstract socket) and then make a decision at that point. Is that what happens? If so, do you think in those rare cases, the dispatcher somehow sleeps through the pipe event? (And if that is what happens, what if the dispatcher wakes itself up every ~5 minutes up to check the pipe proactively? I know this gets uncomfortably close to polling and heartbeating, but all this would need is a way to make sure these dispatcher processes do not run forever. I am thinking about shared clusters like SLURM where hundreds of people share the same login nodes, those nodes keep running for months, and any downtime to clear things out is a major disruption.)

[shikokuchuo]

The very rare crashes would in my experience be due to devtools usage combined with updates to the compiled code in nanonext. If I am just making updates to mirai I don’t experience any crashes – and I regularly install new versions without necessarily resetting daemons first. If there is a difference in experience, then it might be a MacOS thing assuming you are developing on that platform. I would certainly not expect end users to experience any crashes or dispatcher not exiting.

As to how dispatcher exits:

• dispatcher runs a loop in which it waits on a condition variable being signalled on various pipe and message events
• the 'condition variable' is actually more of a semaphore, but note that it contains a flag
• the only event registered to trip the flag is a pipe disconnect on the main socket to host
• once the flag is tripped, execution will break out of the loop and end

Hence, there is no decision to be made whether to exit as you have described - if dispatcher wakes because of a disconnect, the code path is already set for it to exit.

[wlandau]

Thanks for explaining, @shikokuchuo. devtools makes sense as a possible explanation, I will keep an eye on it. And indeed, a slick mechanism for making the dispatcher exit when it should.

How does a condition variable get updated in NNG? If the CV is e.g. on a server dialing into a client, how exactly does the NNG deliver the pipe event from a disconnect? Is there a thread on the client that sends a special group of packets over TCP?

[shikokuchuo]

The condition variable is local. Nothing gets transmitted. The events themselves, i.e. message receive completion or pipe event, signal the CV - and these happen from NNG internal threads. Hence why we can have the main R thread just call wait() on the CV and wait for these things to happen.

The mechanism at dispatcher is the same as that for daemons actually, they both employ a wait and conditional break from the loop. For dispatcher here:

mirai/R/mirai.R

Line 292 in f2b1993

wait(cv) || break

You can have a read of the CV man page - the section 'Flag' for why that works:
https://shikokuchuo.net/nanonext/reference/cv.html

Hope that explains more.

[wlandau]

It does explain more, thanks. I guess what I am wondering about is the stuff that goes on in those NNG threads that tell one end of the socket that the other end of the socket is disconnected. When can we trust the NNG threads to actively send this information?

[shikokuchuo]

That only happens if it is a clean exit, then the TCP protocols require a response to be sent so one side knows the other side is disconnecting. But otherwise, if a socket doesn't get back responses within a certain timeframe than an error will be returned, telling it that it is disconnected. Given the widespread adoption of NNG, I don't believe that the implementation can be flawed, not at such a basic level.