-
Notifications
You must be signed in to change notification settings - Fork 60
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Plugin forking threads for each chunk #155
Comments
In AGRESSIVE mode, all of the chunks are passed to the compiler in one single compilation. The dependency tree is normalized and relationships are clearly defined. In STANDARD mode, each entry point equates to a compilation. In this mode, webpack can duplicate a module into multiple output chunks. Closure Compiler doesn't support this so each chunk tree be compiled individually. Your use case sounds like it's pushing limits here. What would be needed is a thread manager to limit the number of parallel compilations. |
@ChadKillingsworth
We weren't having problems with the v3 plugin though. The previous v3 version used a tuneable concurrency flag to limit the number of parallel compilations performed by the plugin. I think that we need something like async.queue, or some other approach, like a "thread manager" to limit parallel work. |
We're seeing a threading and concurrency problem trying to upgrade our CC webpack plugin.
We've moved from the old v3 version:
https://www.npmjs.com/package/webpack-closure-compiler
To the latest v4 plugin:
https://www.npmjs.com/package/closure-webpack-plugin
We're seeing a pretty wild performance degradation here. The previous v3 version used a tuneable
concurrency
flag to limit the number of parallel compilations performed by the plugin:https://github.com/roman01la/webpack-closure-compiler/blob/3677e5e6672af68eb4608491d39d9e39ba8bf55e/index.js#L99
This would use
async.queue
to only execute CC at a specified concurrency.Testing the new plugin with the default configuration from the docs we see a separate thread spun for every chunk in our webpack compilation. We saw thousands of java threads, and knocked down our development environments.
For additional detail, we tested some different combinations of options. Using any of the
platform
options does not affect the result. We can useplatform: javascript
and then still see hundreds of node threads, as opposed to JVM threads.We also tested switching the
mode
toAGRESSIVE_BUNDLE
. When we run in AGRESSIVE mode though, we do not see more than one compilation at a time. Our compilation is not compatible with that option, but it is worth noting that the problem does not happen when using the AGRESSIVE option.On investigation, it seems like the cause of this explosion in threads might be a bug. The code bifurcates on the bundling type:
closure-webpack-plugin/src/closure-compiler-plugin.js
Lines 448 to 452 in 1246f6e
In the standard codepath, there is a
forEach
over each chunk in the compilation:closure-webpack-plugin/src/closure-compiler-plugin.js
Line 467 in 1246f6e
which pushes the chunk onto a compilations list:
closure-webpack-plugin/src/closure-compiler-plugin.js
Lines 497 to 498 in 1246f6e
And then
Promise.all
s them:closure-webpack-plugin/src/closure-compiler-plugin.js
Line 553 in 1246f6e
But the AGRESSIVE code path only does a single compilation promise:
closure-webpack-plugin/src/closure-compiler-plugin.js
Lines 776 to 777 in 1246f6e
We suspect this explains the explosion in background work, and the difference in behavior between STANDARD and AGRESSIVE.
We're running webpack v4 on node 12 with an otherwise uninteresting configuration. 270 entrypoints, 2500 modules, and about 300 emitted chunks.
The text was updated successfully, but these errors were encountered: