New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cmd/compile/internal/escape: performance issue #67313
Comments
cc @golang/compiler |
Thanks for the report. For enormous generated packages of this sort (800k lines, 95k functions) you are virtually guaranteed to run into compile time disasters of one sort or another; if not in escape analysis, then in some other portion of the back end. Go is an open source project, if you'd like to debug the problem and perhaps send a patch, you're welcome to go ahead with that, but at least from my perspective working on these sorts of issues doesn't seem like a great use of time for the C&R team. The underlying problem here is that the code generator in question is manufacturing a single enormous monolithic package, instead of breaking its output up into chunks or smaller units. This approach simply doesn't scale when inputs get larger and larger. Have you filed an issue with the project that owns the code generator? |
Thanks for the reply.
I totally understand that this is not an average case. I decided to report the issue, because we have two same-sized huge packages with very different escape analysis time. We can close this issue, if that's not the case. |
Go version
go version go1.22.3 linux/amd64
Output of
go env
in your module/workspace:What did you do?
Repo with full repro and pprof profiles: https://github.com/podtserkovskiy/go-slow-escape
We have a Go package generated from a large thrift file
Bad.thrift
bygithub.com/facebook/fbthrift
codegen.The
Bad.thrift
file has ~12k significant lines of code and generated Go package has ~924k significant lines of code.It takes 43 minutes for
cmd/compile
to compile it, which seems to be reasonable until we start to compare it with other files.We have another big thrift file
Good.thrift
, it has ~11k significant lines of code and generated Go package has ~737k significant lines of code.However it takes just 4 minutes to compile this Go package.
The key difference between these files is that
Bad.thrift
structures is more interconnected thanGood.thrift
.The compilation time of
Bad.thrift
expected to be a bit longer thanGood.thrift
, but 39 min difference (10x!) looks suspicious.What did you see happen?
My next step after discovering this facts was adding
-cpuprofile
flag tocmd/compile
and the results look very interesting.The compiler spends ~74% of time in
cmd/compile/internal/escape.Funcs
when processingBad.thrift
.However
Good.thrift
's profile looks completely different, it spends majority of time inssagen.Compile
instead.SVG files for both
Bad.thrift
svg andGood.thrift
svg are attached.The raw pprof profile files are there
good.cpu
andbad.cpu
.What did you expect to see?
That makes me think we hit a performance bottleneck there.
I would be really grateful if somebody familiar with this part of the compiler can take a look on this issue 🙏
The text was updated successfully, but these errors were encountered: