worker_threads: support "eval" mode

[robtweed]

I know that Nodebox is intended for small application/demonstration purposes, and given the practical limitations, I suspect the answer is no, but, given the capabilities of our own products that I'd like to be able to demo in Nodebox, I'm interested to know if, from within the running Nodebox Node.js environment, it is (or will be in future) possible to fire up a child process or Worker Thread.

One even whackier thought: could Nodebox start and access a WebWorker in the browser? Again I suspect not, but if so, it could be a potentially interesting way to emulate a Node.js Worker

BTW - are questions such as these best posted as Discussions or here as Issues?

[DeMoorJasper]

We're currently working on the worker_threads api. I think that one will be released next week.

Child processes is mostly already implemented.

Synchronous spawning/communication is currently not supported as locking a thread requires sharedarraybuffer/atomics and all the compatibility/security issues that come along with that

Creating issues is fine, I should probably just disable discussions

[robtweed]

That will be amazing - I shall await with great interest.

[DeMoorJasper]

Our worker_threads implementation should be live, let us know if you find any issues or have any feedback

[robtweed]

Thanks for adding that capability. I've launched straight in and attempted to run our qoper8-fastify module (https://github.com/robtweed/qoper8-fastify) using a Nodebox adaptation of the demo app you'll see described there.

First thing I found was that if my Nodebox package.json file just had qoper8-fastify as a dependency, then it wouldn't start up which, after a bit of experimentation , was because it didn't seem to have installed Fastify itself, even though Fastify is a dependency in qoper8-fastify's own package.json.

After explicitly adding fastify to the Nodebox package.json (ie in addition to qoper8-fastify), it now loads and the Qoper8-wt queue starts ok.

I'm guessing that you have to handle Fastify's loading in a special way, but I'm sure others may load it as a dependency of their own modules that they may want to use with Nodebox, so it may be something you need to check out?

Anyway, having got it essentially up and running, unfortunately as soon as I send a request, it goes wrong and I get an error returned:

{
statusCode: 500,
error: "Internal Server Error",
message: "(0 , _nodeworkerthreads.Worker) is not a constructor"
}

Now this may be due to something I'm doing wrong in my code - I'll keep searching and debugging, but I thought I'd flag it up now in case you recognise that error and have any thoughts on what might cause it?

[DeMoorJasper]

Sorry about that, should've given an update here, but we rolled back that change for now as it caused some issues with next, we're investigating and will let you know once it's back in production

[robtweed]

No worries - let me know when it's back in action and I'll put it through its paces :-)

[DeMoorJasper]

Worker threads have been merged back into production

[robtweed]

Thanks - I'm just trying it out but I think there's a problem the way I set up Worker Threads. I'm using this syntax:

  let worker = new Worker(workerCode, {eval: true});

ie rather than loading the worker's logic from a file, I use a variable containing the code that is then eval'd.

It seems to work on the face of it - a valid worker object is returned, but nothing in the worker code appears to get invoked when I use worker.postMessage(msg) to send a message object.

One question - should I be able to see the output from console.log(xxx) commands that run within the worker? I certainly don't appear to see anything, but then again if the workerCode wasn't actually loaded, that could explain that too.

I can try loading the code the "conventional" way from a file, but that's not how I'll be wanting to use it with our qoper8-wt module.

Many thanks!
Rob

[kettanaito]

Hey, @robtweed. I've worked on the worker_threads implementation, allow me to shed some light on its behavior.

At the moment, we support a parent/child process relationship with the Worker API. This means you can spawn workers from files, maintaining the communication between them via .postMessage and parentPort. We've tried to implement the worker_threads API to its fullest extent according to the Node.js docs, excluding only moveMessagePortToContext() and markAsUntransferable().

The direct evaluation is not currently supported. I will create a task for it and will advocate for its implementation in the upcoming releases.

[robtweed]

Many thanks for that detailed explanation, @kettanaito ! That's no problem - I'm happy to await implementation of the eval mechanism for Worker Threads in Nodebox. Just wanted to make sure you were aware. Keep up the great work!

[robtweed]

Oh, one thought - just to explain why the eval mechanism is so useful - it allows the development of a packaged-up module without having to worry about the naming and file path needed for a file containing the code that runs in the Worker. So, for example, the logic for my qoper8-wt module is completely self-contained and doesn't need to know anything about the file system used by a user of it.

[kettanaito]

@robtweed, yeah, makes perfect sense. I will see to it so you would also be able to do that in Nodebox.

[arcanis]

On top of the eval issue, I saw the following note:

"Worker.ref()" is not yet implemented. Please file an issue on GitHub if you rely on this feature.

We use Worker#ref in Yarn as part of our worker pool: we have a bunch of workers, all unref'd, and each time we send them a job we call ref to make sure we don't exit before receiving the result. Then we unref them again, and the cycle repeats.