Worryingly high memory use (and leak?) during testing

[freemin7]

┌ Info: System information:
│ Binary dependencies:
│ - NEO_jll: 22.25.23529+0
│ - libigc_jll: 1.0.11378+0
│ - gmmlib_jll: 22.1.3+0
│ - SPIRV_LLVM_Translator_unified_jll: 0.2.0+0
│ - SPIRV_Tools_jll: 2022.1.0+0
│ 
│ Toolchain:
│ - Julia: 1.7.3
│ - LLVM: 12.0.1
│ 
│ 1 driver:
│ - 00000000-0000-0000-1707-a06e01030000 (v1.3.0, API v1.3.0)
│ 
│ 1 device:
└ - Intel(R) UHD Graphics P630 [0x3e96]
                                                  |          | ---------------- CPU ---------------- |
Test                                     (Worker) | Time (s) | GC (s) | GC % | Alloc (MB) | RSS (MB) |
array                                         (2) |     1.60 |   0.04 |  2.5 |     276.05 |   585.97 |
examples                                      (2) |    14.34 |   0.00 |  0.0 |       9.97 |   585.97 |
      From worker 2:	WARNING: Method definition #2699#kernel(Any) in module Main at /home/joto/.julia/packages/oneAPI/mHp15/test/execution.jl:293 overwritten at /home/joto/.julia/packages/oneAPI/mHp15/test/execution.jl:301.
execution                                     (2) |    42.62 |   2.01 |  4.7 |    7578.43 |   745.37 |
level-zero                                    (2) |     2.88 |   0.05 |  1.7 |     306.18 |   745.37 |
pointer                                       (2) |     0.21 |   0.00 |  0.0 |      12.31 |   745.37 |
device/intrinsics                             (2) |    63.98 |   3.81 |  5.9 |   12120.96 |  1145.53 |
gpuarrays/indexing scalar                     (2) |    18.04 |   0.98 |  5.4 |    3239.98 |  1261.28 |
gpuarrays/reductions/reducedim!               (2) |   151.24 |   8.27 |  5.5 |   26046.24 |  2344.89 |
gpuarrays/linalg                              (2) |    74.94 |   3.10 |  4.1 |   12603.09 |  3300.46 |
gpuarrays/math/power                          (2) |    34.64 |   1.88 |  5.4 |    6110.72 |  3511.20 |
gpuarrays/linalg/mul!/vector-matrix           (2) |    88.00 |   3.62 |  4.1 |   13997.27 |  4425.48 |
gpuarrays/indexing multidimensional           (2) |    42.98 |   2.14 |  5.0 |    7570.85 |  4710.65 |
gpuarrays/interface                           (2) |     5.09 |   0.19 |  3.8 |     689.76 |  4757.80 |
gpuarrays/reductions/any all count            (2) |    20.73 |   0.92 |  4.4 |    3385.59 |  4861.70 |
gpuarrays/reductions/minimum maximum extrema  (2) |   231.66 |  10.06 |  4.3 |   34656.54 |  6431.69 |
gpuarrays/uniformscaling                      (2) |     9.47 |   0.34 |  3.6 |    1257.00 |  6431.69 |
gpuarrays/linalg/mul!/matrix-matrix           (2) |   182.76 |   7.28 |  4.0 |   26287.22 |  8479.74 |
gpuarrays/math/intrinsics                     (2) |     4.16 |   0.19 |  4.6 |     685.35 |  8479.74 |
gpuarrays/linalg/norm                         (2) |   362.17 |  12.16 |  3.4 |   38170.87 |  9847.84 |
gpuarrays/statistics                          (2) |   135.30 |   5.44 |  4.0 |   18106.43 | 10524.75 |
gpuarrays/reductions/mapreduce                (2) |   495.31 |  17.34 |  3.5 |   57147.60 | 12348.84 |
gpuarrays/constructors                        (2) |    16.90 |   0.52 |  3.1 |    2413.53 | 12456.36 |
gpuarrays/random                              (2) |    32.64 |   1.39 |  4.3 |    5132.55 | 12456.36 |
gpuarrays/base                                (2) |    61.33 |   2.99 |  4.9 |   10806.98 | 12768.13 |
gpuarrays/reductions/== isequal               (2) |   253.17 |   4.77 |  1.9 |   17205.50 | 13480.29 |
gpuarrays/broadcasting                        (2) |   459.49 |  21.83 |  4.8 |   56494.34 | 17444.71 |
gpuarrays/reductions/mapreducedim!            (2) |   131.21 |   5.64 |  4.3 |   13080.49 | 18137.08 |
gpuarrays/reductions/reduce                   (2) |    39.97 |   0.42 |  1.0 |    2175.47 | 18137.08 |
gpuarrays/reductions/sum prod                 (2) |   590.88 |  34.96 |  5.9 |   42650.34 | 19826.98 |
Testing finished in 1 hour, 1 second, 627 milliseconds

Test Summary: | Pass  Total
  Overall     | 6703   6703
    SUCCESS
     Testing oneAPI tests passed 

shell> ps -o pid,user,vsz,rss,comm,args
    PID USER        VSZ   RSS COMMAND         COMMAND
2538755 joto   11604  5300 bash            -bash
2538807 joto   72336  3084 julia           julia
2538809 joto  1766440 303012 julia         /home/joto/.julia/juliaup/julia-1.7.3+0.x64/bin/julia
2561125 joto    9932  3468 bash            /bin/bash -c (ps -o 'pid,user,vsz,rss,comm,args') && true
2561129 joto   12100  3220 ps              ps -o pid,user,vsz,rss,comm,args

julia> GC.gc()

shell> ps -o pid,user,vsz,rss,comm,args
    PID USER        VSZ   RSS COMMAND         COMMAND
2538755 joto   11604  5300 bash            -bash
2538807 joto   72336  3084 julia           julia
2538809 joto  1670580 285872 julia         /home/joto/.julia/juliaup/julia-1.7.3+0.x64/bin/julia
2561650 joto    9932  3392 bash            /bin/bash -c (ps -o 'pid,user,vsz,rss,comm,args') && true
2561654 joto   12100  3268 ps              ps -o pid,user,vsz,rss,comm,args

[maleadt]

What exactly is the problem? Do you have any evidence this is a memory leak? Julia is a garbage collected language, so unless there's lots of memory pressure it is expected that the physical usage will be higher.

[freemin7]

shouldn't a GC.gc() clean all garbage?

[maleadt]

If you run GC.gc(true) a couple of times, it should. Of course, that's assuming that Level Zero's allocator isn't backed by a pool either (like CUDA.jl stream-ordered allocator is).

[freemin7]

I'd try that again.

[freemin7]

After a lot of GC.gc(true) I am down to 299893 joto 1673772 295064 julia. This does not prove a memory leak itself.