Failing instruction starting at xxxxx wasn't in RVM address space

[qinsoon]

I saw a failed CI run for the PR #105 after merging master with this PR: https://github.com/mmtk/mmtk-jikesrvm/runs/5825456121?check_suite_focus=true. After a rerun, it passed.

I don't think this is related to the PR, and it is very hard to reproduce the issue. I have run the same command for the PR and for the master 896a3dc for 200 runs, and they all succeeded.

Failing instruction starting at xxxxx wasn't in RVM address space usually means a segfault in the Rust code.

./dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/rvm -Xms75M -Xmx75M -jar /home/runner/work/mmtk-jikesrvm/mmtk-jikesrvm/.github/scripts/../../repos/jikesrvm/dacapo/dacapo-2006-10-MR2.jar luindex
===== DaCapo luindex starting =====
adding luindex/william/README
adding luindex/william/comedies/allswellthatendswell
adding luindex/william/comedies/asyoulikeit
adding luindex/william/comedies/comedyoferrors
adding luindex/william/comedies/cymbeline
adding luindex/william/comedies/loveslabourslost
adding luindex/william/comedies/measureforemeasure
adding luindex/william/comedies/merchantofvenice
adding luindex/william/comedies/merrywivesofwindsor
adding luindex/william/comedies/midsummersnightsdream
adding luindex/william/comedies/muchadoaboutnothing
adding luindex/william/comedies/periclesprinceoftyre
JikesRVM: Failing instruction starting at f7cb6910 wasn't in RVM address space
JikesRVM: unexpected hardware trap outside of RVM address space - 0xf7cb6910 0x597fffec
fault address 0xb2ffffe0
eip           0xf7cb6910
eax (T0)      (nil)
ebx (ctrs)    0x597fffec
ecx (S0)      0x597fff05
edx (T1)      0x597ffff0
esi (TR)      0x597fffec
edi (S1)      0x597ffff4
ebp           0x68d097a8
esp (SP)      0x68d09730
cs            0x23
ds            0x2b
es            0x2b
fs            (nil)
gs            0x63
ss            0x2b
trapno        0x0000000e
err           0x00000004
eflags        0x00010246
fpregs        0x5887dc10
oldmask       0x00020000
cr2           0xb2ffffe0
attempting to dump proc map ...
12800000-13800000 rwxp 00000000 00:00 0 
17800000-17c00000 rwxp 00000000 00:00 0 
18000000-18400000 rwxp 00000000 00:00 0 
3ac00000-3b800000 rwxp 00000000 00:00 0 
4e800000-4ec00000 rwxp 00000000 00:00 0 
56633000-56638000 r-xp 00000000 08:01 2358757                            /home/runner/work/mmtk-jikesrvm/mmtk-jikesrvm/repos/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/JikesRVM
56638000-56639000 r-xp 00004000 08:01 2358757                            /home/runner/work/mmtk-jikesrvm/mmtk-jikesrvm/repos/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/JikesRVM
56639000-5663a000 rwxp 00005000 08:01 2358757                            /home/runner/work/mmtk-jikesrvm/mmtk-jikesrvm/repos/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/JikesRVM
57884000-5b970000 rwxp 00000000 00:00 0                                  [heap]
60000000-62177000 rwxp 00000000 08:01 2358685                            /home/runner/work/mmtk-jikesrvm/mmtk-jikesrvm/repos/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/RVM.data.image
64000000-6521a000 rwxp 00000000 08:01 2358681                            /home/runner/work/mmtk-jikesrvm/mmtk-jikesrvm/repos/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/RVM.code.image
67000000-67074000 r-xp 00000000 08:01 2358695                            /home/runner/work/mmtk-jikesrvm/mmtk-jikesrvm/repos/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/RVM.rmap.image
68000000-69c00000 rwxp 00000000 00:00 0 
dd4e8000-dd4e9000 ---p 00000000 00:00 0 
dd4e9000-de4e9000 rwxp 00000000 00:00 0 
de4e9000-de4ea000 ---p 00000000 00:00 0 
de4ea000-df4ea000 rwxp 00000000 00:00 0 
df4ea000-df4eb000 ---p 00000000 00:00 0 
df4eb000-e04eb000 rwxp 00000000 00:00 0 
e04eb000-e04ec000 ---p 00000000 00:00 0 
e04ec000-e14ec000 rwxp 00000000 00:00 0 
e14ec000-e14ed000 ---p 00000000 00:00 0 
e14ed000-e24ed000 rwxp 00000000 00:00 0 
e24ed000-e24ee000 ---p 00000000 00:00 0 
e24ee000-e34ee000 rwxp 00000000 00:00 0 
e34ee000-e34ef000 ---p 00000000 00:00 0 
e34ef000-e44ef000 rwxp 00000000 00:00 0 
e44ef000-e44f0000 ---p 00000000 00:00 0 
e44f0000-e54f0000 rwxp 00000000 00:00 0 
e54f0000-e54f1000 ---p 00000000 00:00 0 
e54f1000-e64f1000 rwxp 00000000 00:00 0 
e64f1000-e64f2000 ---p 00000000 00:00 0 
e64f2000-e74f2000 rwxp 00000000 00:00 0 
e74f2000-e74f3000 ---p 00000000 00:00 0 
e74f3000-e84f3000 rwxp 00000000 00:00 0 
e84f3000-e84f4000 ---p 00000000 00:00 0 
e84f4000-e94f4000 rwxp 00000000 00:00 0 
e94f4000-e94f5000 ---p 00000000 00:00 0 
e94f5000-ea4f5000 rwxp 00000000 00:00 0 
ea4f5000-ea4f6000 ---p 00000000 00:00 0 
ea4f6000-eb4f6000 rwxp 00000000 00:00 0 
eb4f6000-eb4f7000 ---p 00000000 00:00 0 
eb4f7000-ec4f7000 rwxp 00000000 00:00 0 
ec4f7000-ec4f8000 ---p 00000000 00:00 0 
ec4f8000-ed4f8000 rwxp 00000000 00:00 0 
ed4f8000-ed4f9000 ---p 00000000 00:00 0 
ed4f9000-ee4f9000 rwxp 00000000 00:00 0 
ee4f9000-ee4fa000 ---p 00000000 00:00 0 
ee4fa000-ef4fa000 rwxp 00000000 00:00 0 
ef4fa000-ef4fb000 ---p 00000000 00:00 0 
ef4fb000-f04fb000 rwxp 00000000 00:00 0 
f04fb000-f04fc000 ---p 00000000 00:00 0 
f04fc000-f14fc000 rwxp 00000000 00:00 0 
f14fc000-f14fd000 ---p 00000000 00:00 0 
f14fd000-f24fd000 rwxp 00000000 00:00 0 
f24fd000-f24fe000 ---p 00000000 00:00 0 
f24fe000-f34fe000 rwxp 00000000 00:00 0 
f34fe000-f34ff000 ---p 00000000 00:00 0 
f34ff000-f44ff000 rwxp 00000000 00:00 0 
f44ff000-f4500000 ---p 00000000 00:00 0 
f4500000-f5500000 rwxp 00000000 00:00 0 
f5500000-f55ae000 rwxp 00000000 00:00 0 
f55ae000-f5600000 ---p 00000000 00:00 0 
f5600000-f56ff000 rwxp 00000000 00:00 0 
f56ff000-f5700000 ---p 00000000 00:00 0 
f5700000-f57d7000 rwxp 00000000 00:00 0 
f57d7000-f5800000 ---p 00000000 00:00 0 
f58af000-f58bf000 r-xp 00000000 08:01 2358755                            /home/runner/work/mmtk-jikesrvm/mmtk-jikesrvm/repos/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjavanio.so
f58bf000-f58c0000 r-xp 0000f000 08:01 2358755                            /home/runner/work/mmtk-jikesrvm/mmtk-jikesrvm/repos/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjavanio.so
f58c0000-f58c1000 rwxp 00010000 08:01 2358755                            /home/runner/work/mmtk-jikesrvm/mmtk-jikesrvm/repos/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjavanio.so
f58c1000-f58c2000 rwxp 00000000 00:00 0 
f58c2000-f58dc000 r-xp 00000000 08:01 2358751                            /home/runner/work/mmtk-jikesrvm/mmtk-jikesrvm/repos/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjavalang.so
f58dc000-f58dd000 r-xp 00019000 08:01 2358751                            /home/runner/work/mmtk-jikesrvm/mmtk-jikesrvm/repos/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjavalang.so
f58dd000-f58de000 rwxp 0001a000 08:01 2358751                            /home/runner/work/mmtk-jikesrvm/mmtk-jikesrvm/repos/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjavalang.so
f58de000-f58df000 rwxp 00000000 00:00 0 
f58df000-f58e0000 ---p 00000000 00:00 0 
f58e0000-f68e0000 rwxp 00000000 00:00 0 
f68e0000-f68e1000 ---p 00000000 00:00 0 
f68e1000-f7b24000 rwxp 00000000 00:00 0 
f7b24000-f7bed000 r-xp 00000000 08:01 4902922                            /lib32/libm-2.27.so
f7bed000-f7bee000 r-xp 000c8000 08:01 4902922                            /lib32/libm-2.27.so
f7bee000-f7bef000 rwxp 000c9000 08:01 4902922                            /lib32/libm-2.27.so
f7bef000-f7c0a000 r-xp 00000000 08:01 4902932                            /lib32/libpthread-2.27.so
f7c0a000-f7c0b000 r-xp 0001a000 08:01 4902932                            /lib32/libpthread-2.27.so
f7c0b000-f7c0c000 rwxp 0001b000 08:01 4902932                            /lib32/libpthread-2.27.so
f7c0c000-f7c0e000 rwxp 00000000 00:00 0 
f7c0e000-f7c16000 r-xp 00000000 08:01 4902934                            /lib32/librt-2.27.so
f7c16000-f7c17000 r-xp 00007000 08:01 4902934                            /lib32/librt-2.27.so
f7c17000-f7c18000 rwxp 00008000 08:01 4902934                            /lib32/librt-2.27.so
f7c18000-f7c34000 r-xp 00000000 08:01 3622477                            /usr/lib32/libgcc_s.so.1
f7c34000-f7c35000 r-xp 0001b000 08:01 3622477                            /usr/lib32/libgcc_s.so.1
f7c35000-f7c36000 rwxp 0001c000 08:01 3622477                            /usr/lib32/libgcc_s.so.1
f7c36000-f7c39000 r-xp 00000000 08:01 4902921                            /lib32/libdl-2.27.so
f7c39000-f7c3a000 r-xp 00002000 08:01 4902921                            /lib32/libdl-2.27.so
f7c3a000-f7c3b000 rwxp 00003000 08:01 4902921                            /lib32/libdl-2.27.so
f7c3b000-f7e0d000 r-xp 00000000 08:01 4902918                            /lib32/libc-2.27.so
f7e0d000-f7e0e000 ---p 001d2000 08:01 4902918                            /lib32/libc-2.27.so
f7e0e000-f7e10000 r-xp 001d2000 08:01 4902918                            /lib32/libc-2.27.so
f7e10000-f7e11000 rwxp 001d4000 08:01 4902918                            /lib32/libc-2.27.so
f7e11000-f7e14000 rwxp 00000000 00:00 0 
f7e17000-f7e21000 r-xp 00000000 08:01 2358750                            /home/runner/work/mmtk-jikesrvm/mmtk-jikesrvm/repos/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjavaio.so
f7e21000-f7e22000 r-xp 00009000 08:01 2358750                            /home/runner/work/mmtk-jikesrvm/mmtk-jikesrvm/repos/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjavaio.so
f7e22000-f7e23000 rwxp 0000a000 08:01 2358750                            /home/runner/work/mmtk-jikesrvm/mmtk-jikesrvm/repos/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjavaio.so
f7e23000-f7e24000 rwxp 00000000 00:00 0 
f7e24000-f7e25000 r-xp 00000000 08:01 2358759                            /home/runner/work/mmtk-jikesrvm/mmtk-jikesrvm/repos/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjvm_jni.so
f7e25000-f7e26000 r-xp 00000000 08:01 2358759                            /home/runner/work/mmtk-jikesrvm/mmtk-jikesrvm/repos/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjvm_jni.so
f7e26000-f7e27000 rwxp 00001000 08:01 2358759                            /home/runner/work/mmtk-jikesrvm/mmtk-jikesrvm/repos/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjvm_jni.so
f7e27000-f7f67000 r-xp 00000000 08:01 2358760                            /home/runner/work/mmtk-jikesrvm/mmtk-jikesrvm/repos/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libmmtk.so
f7f67000-f7f68000 ---p 00140000 08:01 2358760                            /home/runner/work/mmtk-jikesrvm/mmtk-jikesrvm/repos/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libmmtk.so
f7f68000-f7f71000 r-xp 00140000 08:01 2358760                            /home/runner/work/mmtk-jikesrvm/mmtk-jikesrvm/repos/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libmmtk.so
f7f71000-f7f72000 rwxp 00149000 08:01 2358760                            /home/runner/work/mmtk-jikesrvm/mmtk-jikesrvm/repos/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libmmtk.so
f7f72000-f7f73000 rwxp 00000000 00:00 0 
f7f73000-f7f89000 r-xp 00000000 08:01 2358758                            /home/runner/work/mmtk-jikesrvm/mmtk-jikesrvm/repos/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjvm.so
f7f89000-f7f8a000 r-xp 00015000 08:01 2358758                            /home/runner/work/mmtk-jikesrvm/mmtk-jikesrvm/repos/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjvm.so
f7f8a000-f7f8b000 rwxp 00016000 08:01 2358758                            /home/runner/work/mmtk-jikesrvm/mmtk-jikesrvm/repos/jikesrvm/dist/RFastAdaptiveMarkSweep_x86_64_m32-linux/libjvm.so
f7f8b000-f7f8d000 rwxp 00000000 00:00 0 
f7f8d000-f7f90000 r--p 00000000 00:00 0                                  [vvar]
f7f90000-f7f91000 r-xp 00000000 00:00 0                                  [vdso]
f7f91000-f7fb7000 r-xp 00000000 08:01 4902914                            /lib32/ld-2.27.so
f7fb7000-f7fb8000 r-xp 00025000 08:01 4902914                            /lib32/ld-2.27.so
f7fb8000-f7fb9000 rwxp 00026000 08:01 4902914                            /lib32/ld-2.27.so
ffdea000-ffe0c000 rwxp 00000000 00:00 0                                  [stack]

[qinsoon]

I also see this in #109. It happened for mark sweep as well. I was not able to reproduce it on my development machine. And it happened frequently enough to be a problem for us.

So this possibly is an issue from v0.11 (or the commits before v0.11). My guess is that this could be related with malloc (as I only saw this issue in our mark sweep plan). Possibly on the Github CI runner, malloc gives us unexpected addresses, or addresses in an unexpected pattern. The issue is segfault, so possibly we should look into unsafe code in our malloc marksweep.

[wks]

Similar errors show up in the hsqldb benchmark multiple times recently.

• https://github.com/mmtk/mmtk-core/actions/runs/6582446394/job/17883971951?pr=988
• https://github.com/mmtk/mmtk-jikesrvm/actions/runs/6627879048/job/18023992683?pr=150

[wks]

This still happens. https://github.com/mmtk/mmtk-core/actions/runs/7755790420/job/21151876789?pr=1067

And it is not related to ASLR. When I ran JikesRVM with setarch -R, this still occurs.

[wks]

Now when running hsqldb with RFastAdaptiveMarkSweep, there is a 1/10 chance to trigger this error. Currently my clues are:

• It is not related to ASLR. It is reproducible with setarch -R.
• It is only reproducible with RFastAdaptiveMarkSweep. I haven't reproduced it with RBaseBaseMarkSweep or RFastAdaptiveSemiSpace.
• It is not related to the number of GC threads. It can be reproduced with MMTK_THREADS=1.
• It doesn't seem to be related to GC. I increased the heap size to 4G (-Xms4000M -Xmx4000M), and it is even more likely to trigger this problem.