Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix sigabrt in parfors tests #9568

Merged
merged 2 commits into from May 13, 2024
Merged

Conversation

esc
Copy link
Member

@esc esc commented May 10, 2024

This fixes a test, where a non-thread safe container is written to during testing

To reproduce on at least linux-64 and osx-arm64 (and probably others):

NUMBA_THREADING_LAYER=workqueue SUBPROC_TEST=1 ./runtests.py -m 32 numba.tests.test_parfors.TestPrangeSpecific.test_tuple_hoisting

On linux-64, this can be debugged with gdb:

(gdb) bt
0  0x00007ffff7c8018b in raise () from /lib/x86_64-linux-gnu/libc.so.6
1  0x00007ffff7c5f859 in abort () from /lib/x86_64-linux-gnu/libc.so.6
2  0x00007ffff7cca3ee in ?? () from /lib/x86_64-linux-gnu/libc.so.6
3  0x00007ffff7cd247c in ?? () from /lib/x86_64-linux-gnu/libc.so.6
4  0x00007ffff7cd412c in ?? () from /lib/x86_64-linux-gnu/libc.so.6
5  0x00007ffff7cd6105 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
6  0x00007ffff7cd82d6 in realloc () from /lib/x86_64-linux-gnu/libc.so.6
7  0x00007fffe706de21 in NRT_Reallocate (ptr=0x224aef0, size=2552489952) at numba/core/runtime/nrt.cpp:539
8  0x00007fffe706dcf6 in NRT_MemInfo_varsize_realloc (mi=0x1fe9580, size=2552489952) at numba/core/runtime/nrt.cpp:498
9  0x00007fffdcabde0d in _3cdynamic_3e::__numba_parfor_gufunc_0x7fffdc60eb20[abi:v19][abi:c8tJTC_2fWQAliW1xhDEoY6EEMEUOEMISPGsAQMVj4QniQ4IXKQEMXwoMGLoQDDVsQR1NHAZtvoQrhyQ_2fKR8sTqKIYOQAmjYgkW7ADge6ERATM1UUQpZoA](Array<unsigned long long, 1, C, mutable, aligned>, list_28Tuple_28DictType_5bint64_2cfloat64_5d_3civ_3dNone_3e_2c_20array_28float64_2c_201d_2c_20C_29_29_29_3civ_3dNone_3e) (sched=...,
   closure____locals______listcomp____v15____v2build__list__0=...) at <string>:4438
10 0x00007fffdcab624e in __gufunc__._ZN13_3cdynamic_3e36__numba_parfor_gufunc_0x7fffdc60eb20B3v19B120c8tJTC_2fWQAliW1xhDEoY6EEMEUOEMISPGsAQMVj4QniQ4IXKQEMXwoMGLoQDDVsQR1NHAZtvoQrhyQ_2fKR8sTqKIYOQAmjYgkW7ADge6ERATM1UUQpZoAE5ArrayIyLi1E1C7mutable7alignedE119list_28Tuple_28DictType_5bint64_2cfloat64_5d_3civ_3dNone_3e_2c_20array_28float64_2c_201d_2c_20C_29_29_29_3civ_3dNone_3e ()
11 0x00007fffdc926a3b in thread_worker (arg=0x1bd49c0) at numba/np/ufunc/workqueue.c:567
12 0x00007ffff7f8f609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
13 0x00007ffff7d5c293 in clone () from /lib/x86_64-linux-gnu/libc.so.6

On osx-arm64 we can use lldb:

(lldb) run runtests.py -m 32 numba.tests.test_parfors.TestPrangeSpecific.test_tuple_hoisting
Process 13575 launched: '/Users/esc/miniconda3-arm64/envs/numba_3.9/bin/python3' (arm64)
Parallel: 0. Serial: 1
python3(13575,0x17025b000) malloc: Non-aligned pointer 0x600000256880 being freed (2)
python3(13575,0x17025b000) malloc: *** set a breakpoint in malloc_error_break to debug
Process 13575 stopped
* thread #18, stop reason = signal SIGABRT
    frame #0: 0x000000019c35c704 libsystem_kernel.dylib`__pthread_kill + 8
libsystem_kernel.dylib`:
->  0x19c35c704 <+8>:  b.lo   0x19c35c724               ; <+40>
    0x19c35c708 <+12>: pacibsp
    0x19c35c70c <+16>: stp    x29, x30, [sp, #-0x10]!
    0x19c35c710 <+20>: mov    x29, sp
Target 0: (python3) stopped.
(lldb) bt
* thread #18, stop reason = signal SIGABRT
  * frame #0: 0x000000019c35c704 libsystem_kernel.dylib`__pthread_kill + 8
    frame #1: 0x000000019c393c28 libsystem_pthread.dylib`pthread_kill + 288
    frame #2: 0x000000019c2a1ae8 libsystem_c.dylib`abort + 180
    frame #3: 0x000000019c1c2e28 libsystem_malloc.dylib`malloc_vreport + 908
    frame #4: 0x000000019c1d95d4 libsystem_malloc.dylib`malloc_zone_error + 104
    frame #5: 0x000000019c1ca620 libsystem_malloc.dylib`_szone_free + 628
    frame #6: 0x000000019c1b87f4 libsystem_malloc.dylib`nanov2_realloc + 356
    frame #7: 0x000000019c1b85a4 libsystem_malloc.dylib`malloc_zone_realloc + 112
    frame #8: 0x000000019c1b7110 libsystem_malloc.dylib`realloc + 388
    frame #9: 0x000000013a9ff0f8 _nrt_python.cpython-39-darwin.so`NRT_MemInfo_varsize_realloc + 60
    frame #10: 0x000000013d4f41e0
    frame #11: 0x000000019c393fa8 libsystem_pthread.dylib`_pthread_start + 148

This fixes a test, where a non-thread safe container is written to
during testing

To reproduce on at least `linux-64` and `osx-arm64` (and probably
others):

```
NUMBA_THREADING_LAYER=workqueue SUBPROC_TEST=1 ./runtests.py -m 32 numba.tests.test_parfors.TestPrangeSpecific.test_tuple_hoisting
```

On `linux-64`, this can be debugged with `gdb`:

```
(gdb) bt
0  0x00007ffff7c8018b in raise () from /lib/x86_64-linux-gnu/libc.so.6
1  0x00007ffff7c5f859 in abort () from /lib/x86_64-linux-gnu/libc.so.6
2  0x00007ffff7cca3ee in ?? () from /lib/x86_64-linux-gnu/libc.so.6
3  0x00007ffff7cd247c in ?? () from /lib/x86_64-linux-gnu/libc.so.6
4  0x00007ffff7cd412c in ?? () from /lib/x86_64-linux-gnu/libc.so.6
5  0x00007ffff7cd6105 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
6  0x00007ffff7cd82d6 in realloc () from /lib/x86_64-linux-gnu/libc.so.6
7  0x00007fffe706de21 in NRT_Reallocate (ptr=0x224aef0, size=2552489952) at numba/core/runtime/nrt.cpp:539
8  0x00007fffe706dcf6 in NRT_MemInfo_varsize_realloc (mi=0x1fe9580, size=2552489952) at numba/core/runtime/nrt.cpp:498
9  0x00007fffdcabde0d in _3cdynamic_3e::__numba_parfor_gufunc_0x7fffdc60eb20[abi:v19][abi:c8tJTC_2fWQAliW1xhDEoY6EEMEUOEMISPGsAQMVj4QniQ4IXKQEMXwoMGLoQDDVsQR1NHAZtvoQrhyQ_2fKR8sTqKIYOQAmjYgkW7ADge6ERATM1UUQpZoA](Array<unsigned long long, 1, C, mutable, aligned>, list_28Tuple_28DictType_5bint64_2cfloat64_5d_3civ_3dNone_3e_2c_20array_28float64_2c_201d_2c_20C_29_29_29_3civ_3dNone_3e) (sched=...,
   closure____locals______listcomp____v15____v2build__list__0=...) at <string>:4438
10 0x00007fffdcab624e in __gufunc__._ZN13_3cdynamic_3e36__numba_parfor_gufunc_0x7fffdc60eb20B3v19B120c8tJTC_2fWQAliW1xhDEoY6EEMEUOEMISPGsAQMVj4QniQ4IXKQEMXwoMGLoQDDVsQR1NHAZtvoQrhyQ_2fKR8sTqKIYOQAmjYgkW7ADge6ERATM1UUQpZoAE5ArrayIyLi1E1C7mutable7alignedE119list_28Tuple_28DictType_5bint64_2cfloat64_5d_3civ_3dNone_3e_2c_20array_28float64_2c_201d_2c_20C_29_29_29_3civ_3dNone_3e ()
11 0x00007fffdc926a3b in thread_worker (arg=0x1bd49c0) at numba/np/ufunc/workqueue.c:567
12 0x00007ffff7f8f609 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
13 0x00007ffff7d5c293 in clone () from /lib/x86_64-linux-gnu/libc.so.6
```

On `osx-arm64` we can use `lldb`:

```
(lldb) run runtests.py -m 32 numba.tests.test_parfors.TestPrangeSpecific.test_tuple_hoisting
Process 13575 launched: '/Users/esc/miniconda3-arm64/envs/numba_3.9/bin/python3' (arm64)
Parallel: 0. Serial: 1
python3(13575,0x17025b000) malloc: Non-aligned pointer 0x600000256880 being freed (2)
python3(13575,0x17025b000) malloc: *** set a breakpoint in malloc_error_break to debug
Process 13575 stopped
* thread numba#18, stop reason = signal SIGABRT
    frame #0: 0x000000019c35c704 libsystem_kernel.dylib`__pthread_kill + 8
libsystem_kernel.dylib`:
->  0x19c35c704 <+8>:  b.lo   0x19c35c724               ; <+40>
    0x19c35c708 <+12>: pacibsp
    0x19c35c70c <+16>: stp    x29, x30, [sp, #-0x10]!
    0x19c35c710 <+20>: mov    x29, sp
Target 0: (python3) stopped.
(lldb) bt
* thread numba#18, stop reason = signal SIGABRT
  * frame #0: 0x000000019c35c704 libsystem_kernel.dylib`__pthread_kill + 8
    frame #1: 0x000000019c393c28 libsystem_pthread.dylib`pthread_kill + 288
    frame #2: 0x000000019c2a1ae8 libsystem_c.dylib`abort + 180
    frame #3: 0x000000019c1c2e28 libsystem_malloc.dylib`malloc_vreport + 908
    frame #4: 0x000000019c1d95d4 libsystem_malloc.dylib`malloc_zone_error + 104
    frame #5: 0x000000019c1ca620 libsystem_malloc.dylib`_szone_free + 628
    frame #6: 0x000000019c1b87f4 libsystem_malloc.dylib`nanov2_realloc + 356
    frame #7: 0x000000019c1b85a4 libsystem_malloc.dylib`malloc_zone_realloc + 112
    frame numba#8: 0x000000019c1b7110 libsystem_malloc.dylib`realloc + 388
    frame numba#9: 0x000000013a9ff0f8 _nrt_python.cpython-39-darwin.so`NRT_MemInfo_varsize_realloc + 60
    frame numba#10: 0x000000013d4f41e0
    frame numba#11: 0x000000019c393fa8 libsystem_pthread.dylib`_pthread_start + 148
```
@esc esc added this to the 0.60.0-rc1 milestone May 10, 2024
@esc
Copy link
Member Author

esc commented May 10, 2024

PR adds itself to changelog.

@esc esc added the skip_release_notes Skip towncrier requirement label May 10, 2024
@esc esc closed this May 12, 2024
@esc esc reopened this May 12, 2024
@esc esc merged commit dd5c5cb into numba:release0.60 May 13, 2024
21 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants