Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make post processing faster #2561

Open
JoeWang1127 opened this issue Mar 13, 2024 · 2 comments
Open

Make post processing faster #2561

JoeWang1127 opened this issue Mar 13, 2024 · 2 comments
Labels
hermetic-build priority: p2 Moderately-important priority. Fix may not be included in next release. type: process A process-related concern. May include testing, release, or the like.

Comments

@JoeWang1127
Copy link
Collaborator

JoeWang1127 commented Mar 13, 2024

I noticed that post processing a library takes longer than generating it. We should optimize it.

This issue contains two parts:

  1. Identify the slow commands in post processing
  2. Optimize the slow commands.

More context: I don't know which command is slow but I noticed stuck when running the post processor.

@JoeWang1127 JoeWang1127 added priority: p2 Moderately-important priority. Fix may not be included in next release. hermetic-build labels Mar 13, 2024
@diegomarquezp
Copy link
Contributor

From internal discussion:
One nice to have that may benefit us is to stream the output of the commands in real time. Now, the python script blocks until calls to scripts such as generate_library or postprocess_library are finished and then prints the resulting output. It should be a straightforward thing to do

@diegomarquezp
Copy link
Contributor

We may benefit from postprocess_monorepo instead of postprocess_library regarding the overhead of setting up each one of the commands used in postprocessing. For example there is a mvn fmt:format ran for each library, or the fact that copy-code can process more than one yaml in a single run

@zhumin8 zhumin8 added the type: process A process-related concern. May include testing, release, or the like. label Mar 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
hermetic-build priority: p2 Moderately-important priority. Fix may not be included in next release. type: process A process-related concern. May include testing, release, or the like.
Projects
None yet
Development

No branches or pull requests

3 participants