Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Multithreading #9

Open
squattingmonk opened this issue Nov 5, 2019 · 1 comment
Open

[Feature] Multithreading #9

squattingmonk opened this issue Nov 5, 2019 · 1 comment
Labels
enhancement New feature or request
Milestone

Comments

@squattingmonk
Copy link
Owner

Nasher is currently single-threaded. While it is still fast because of the underlying tools, multithreading would increase the speed at which large modules could be processed.

@squattingmonk squattingmonk added the enhancement New feature or request label Nov 5, 2019
@WilliamDraco
Copy link
Contributor

WilliamDraco commented Nov 9, 2019

I had an initial exploration of this. I stuck with the unpack module since I knew it best. A few thoughts:

  • gffConvert appears to be the most appropriate process to parallelise, being repeatedly called over many files.
  • This requires gffConvert to be {.gcsafe.}. This traces through gffToJson/JsonToGff -> fatal -> error -> display* -> displayLine ->display Category*
    Both of those procs with * above reference the cli object, which is a global and therefore blocking {.gcsafe.}. I temporarily replaced fatal and info (which also has a similar proc path) with echo just to test.
  • Spawning many gffConverts worked, and put the CPU to 100% and begins to convert at a high pace, however unfortunately something always snags. Most often I got was parseJson errors, I imagine related to the buffer being overwritten by another thread?

That's just my run of experimentation at the moment.

@squattingmonk squattingmonk modified the milestones: 2.0, 1.0 Oct 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants