[Help Needed] How to Accelerate pprof -top Command Execution for Profiling Data?

[PengWei97]

What version of pprof are you using?

(moose) pw-moose@pwmoose-PowerEdge-T640:~/go/bin$ go version
go version go1.18.1 linux/amd64

If you run pprof from GitHub, what's the Git revision?

Today, I used go install github.com/google/pprof@latest to install, so I think it is the latest version of pprof, but I cannot get the specific version number.

What operating system and processor architecture are you using?

(moose) pw-moose@pwmoose-PowerEdge-T640:~$ uname -a
Linux pwmoose-PowerEdge-T640 6.5.0-28-generic #29~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Apr  4 14:39:20 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

What did you do?

I am currently working with profiling data generated by the MOOSE simulation framework, specifically using the pprof tool to analyze performance. I used the following command to generate profiling data with 32 processes:

MOOSE_PROFILE_BASE=run3_ mpiexec -n 32 ~/projocts/panda/panda-oprof -i s2_bicrystal_cp_elastic3.i > 01.log

This command generated 32 profiling data files named run3_[0-31].prof. I am trying to generate a performance report using the following command:

pprof -top ~/projocts/panda/panda-oprof run3_1.prof

prof_and_oprof.zip

What did you see instead?

However, I am facing an issue where this command takes an unexpectedly long time to execute; it has been over 30 minutes without yielding any results.

What did you expect to see?

I would appreciate any insights or suggestions on how to accelerate the execution of this command. Are there specific flags or methods within pprof that can be utilized to optimize the processing time for large profiling data files?

Thank you for your help and looking forward to your suggestions.

Best regards,

Wei

[aalexand]

I cannot reproduce this problem. The below command executes in a couple seconds for me.

$ pprof -top ./panda-oprof ./run3_0.prof
Local symbolization failed for libmoose-oprof.so.0.0.0: stat /home/pw-moose/projocts/moose/framework/libmoose-oprof.so.0.0.0: no such file or directory
Local symbolization failed for libpanda-oprof.so.0.0.0: stat /home/pw-moose/projocts/panda/lib/libpanda-oprof.so.0.0.0: no such file or directory
Local symbolization failed for libsolid_mechanics-oprof.so.0.0.0: stat /home/pw-moose/projocts/moose/modules/solid_mechanics/lib/libsolid_mechanics-oprof.so.0.0.0: no such file or directory
Local symbolization failed for libmesh_oprof.so.0.0.0: stat /home/pw-moose/miniforge/envs/moose/libmesh/lib/libmesh_oprof.so.0.0.0: no such file or directory
Local symbolization failed for libpetsc.so.3.20.3: stat /home/pw-moose/miniforge/envs/moose/lib/libpetsc.so.3.20.3: no such file or directory
Local symbolization failed for libmpi.so.12.2.2: stat /home/pw-moose/miniforge/envs/moose/lib/libmpi.so.12.2.2: no such file or directory
Local symbolization failed for libtcmalloc_and_profiler.so.4.6.11: stat /home/pw-moose/gperftools/installed/lib/libtcmalloc_and_profiler.so.4.6.11: no such file or directory
Local symbolization failed for libHYPRE-2.29.0.so: stat /home/pw-moose/miniforge/envs/moose/lib/libHYPRE-2.29.0.so: no such file or directory
Local symbolization failed for libphase_field-oprof.so.0.0.0: stat /home/pw-moose/projocts/moose/modules/phase_field/lib/libphase_field-oprof.so.0.0.0: no such file or directory
Local symbolization failed for libstdc++.so.6.0.32: stat /home/pw-moose/miniforge/envs/moose/lib/libstdc++.so.6.0.32: no such file or directory
Local symbolization failed for libhit-oprof.so.0.0.0: stat /home/pw-moose/projocts/moose/framework/contrib/hit/libhit-oprof.so.0.0.0: no such file or directory
Local symbolization failed for libhdf5.so.200.1.0: stat /home/pw-moose/miniforge/envs/moose/lib/libhdf5.so.200.1.0: no such file or directory
Local symbolization failed for 66cd8ad14093e0ad.so: stat /home/pw-moose/projocts/panda/simulation/prm3/step01_elastic_energy/.jitcache/66cd8ad14093e0ad.so: no such file or directory
Local symbolization failed for libgfortran.so.5.0.0: stat /home/pw-moose/miniforge/envs/moose/lib/libgfortran.so.5.0.0: no such file or directory
Local symbolization failed for libgomp.so.1.0.0: stat /home/pw-moose/miniforge/envs/moose/lib/libgomp.so.1.0.0: no such file or directory
Some binary filenames not available. Symbolization may be incomplete.
Try setting PPROF_BINARY_PATH to the search path for local binaries.
File: panda-oprof
Type: cpu
Showing nodes accounting for 41.32s, 98.52% of 41.94s total
Dropped 34 nodes (cum <= 0.21s)
      flat  flat%   sum%        cum   cum%
    25.24s 60.18% 60.18%     25.25s 60.21%  [libmpi.so.12.2.2]
     7.21s 17.19% 77.37%     13.05s 31.12%  [libmoose-oprof.so.0.0.0]
     2.12s  5.05% 82.43%      8.24s 19.65%  [libpanda-oprof.so.0.0.0]
     1.61s  3.84% 86.27%      1.64s  3.91%  [libHYPRE-2.29.0.so]
     1.03s  2.46% 88.72%      1.03s  2.46%  __tan_fma4
     0.85s  2.03% 90.75%     12.57s 29.97%  [libmesh_oprof.so.0.0.0]
     0.71s  1.69% 92.44%      0.71s  1.69%  gethostbyaddr_r
     0.55s  1.31% 93.75%      0.55s  1.31%  [libc.so.6]
     0.55s  1.31% 95.06%      0.57s  1.36%  [libstdc++.so.6.0.32]
     0.38s  0.91% 95.97%      0.38s  0.91%  [libtcmalloc_and_profiler.so.4.6.11]
     0.32s  0.76% 96.73%      8.83s 21.05%  [libsolid_mechanics-oprof.so.0.0.0]
     0.27s  0.64% 97.38%     12.43s 29.64%  [libpetsc.so.3.20.3]
     0.27s  0.64% 98.02%      0.35s  0.83%  [libphase_field-oprof.so.0.0.0]
     0.21s   0.5% 98.52%      0.21s   0.5%  argp_default_parser
         0     0% 98.52%      0.88s  2.10%  Moose::main
         0     0% 98.52%      0.88s  2.10%  main
         0     0% 98.52%      0.88s  2.10%  read_conf_file

[aalexand]

Closing. We can reopen once there's a reproducer.