Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replaying a trace that calls mmap causes an assertion failure #3635

Open
firelizzard18 opened this issue Oct 24, 2023 · 22 comments
Open

Replaying a trace that calls mmap causes an assertion failure #3635

firelizzard18 opened this issue Oct 24, 2023 · 22 comments

Comments

@firelizzard18
Copy link

Given the following program:

package main

import (
	"os"

	"github.com/edsrzf/mmap-go"
)

func main() {
	f, err := os.CreateTemp("", "")
	must(err)
	defer f.Close()
	must(f.Truncate(1<<12))
	x, err := mmap.Map(f, mmap.RDWR, 0) // <-- break here
	must(err)
	defer x.Unmap()
}

func must(err error) {
	if err != nil {
		panic(err)
	}
}

If I place a breakpoint at the call to mmap.Map and replay the trace, rr fails an assertion when I step.

trace
Detaching and terminating target process
dlv dap (149574) exited with code: 0
Starting: /home/firelizzard/go/bin/dlv dap --listen=127.0.0.1:44051 --log-dest=3 from /tmp/foo
DAP server listening at: 127.0.0.1:44051
Type 'dlv help' for list of commands.
[FATAL src/AutoRemoteSyscalls.cc:734:check_syscall_result()] 
 (task 150151 (rec:150135) at time 443)
 -> Assertion `false' failed to hold. Syscall openat failed with errno EACCES opening /proc/150136/fd/22
Tail of trace dump:
{
  real_time:15653.434285 global_time:423, event:`SYSCALL: fcntl' (state:EXITING_SYSCALL) tid:150135, ticks:1509
rax:0x8002 rbx:0x4 rcx:0xffffffffffffffff rdx:0x0 rsi:0x3 rdi:0x4 rbp:0xc00006c898 rsp:0xc00006c858 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006c930 r13:0xc000046400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0xc000046490 gs_base:0x0
}
{
  real_time:15653.434298 global_time:424, event:`SYSCALL: fcntl' (state:ENTERING_SYSCALL) tid:150135, ticks:1551
rax:0xffffffffffffffda rbx:0x4 rcx:0xffffffffffffffff rdx:0x8802 rsi:0x4 rdi:0x4 rbp:0xc00006c898 rsp:0xc00006c858 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006c930 r13:0xc000046400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0xc000046490 gs_base:0x0
}
{
  real_time:15653.434307 global_time:425, event:`SYSCALL: fcntl' (state:EXITING_SYSCALL) tid:150135, ticks:1551
rax:0x0 rbx:0x4 rcx:0xffffffffffffffff rdx:0x8802 rsi:0x4 rdi:0x4 rbp:0xc00006c898 rsp:0xc00006c858 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006c930 r13:0xc000046400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0xc000046490 gs_base:0x0
}
{
  real_time:15653.434320 global_time:426, event:`SYSCALL: epoll_create1' (state:ENTERING_SYSCALL) tid:150135, ticks:1604
rax:0xffffffffffffffda rbx:0x80000 rcx:0xffffffffffffffff rdx:0x0 rsi:0x0 rdi:0x80000 rbp:0xc00006c7d0 rsp:0xc00006c790 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006c8e0 r13:0xc000046400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x123 fs_base:0xc000046490 gs_base:0x0
}
{
  real_time:15653.434336 global_time:427, event:`SYSCALL: epoll_create1' (state:EXITING_SYSCALL) tid:150135, ticks:1604
rax:0x5 rbx:0x80000 rcx:0xffffffffffffffff rdx:0x0 rsi:0x0 rdi:0x80000 rbp:0xc00006c7d0 rsp:0xc00006c790 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006c8e0 r13:0xc000046400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x123 fs_base:0xc000046490 gs_base:0x0
}
{
  real_time:15653.434350 global_time:428, event:`SYSCALL: pipe2' (state:ENTERING_SYSCALL) tid:150135, ticks:1607
rax:0xffffffffffffffda rbx:0x0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x80800 rdi:0xc00006c7c0 rbp:0xc00006c7d0 rsp:0xc00006c7b0 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006c8e0 r13:0xc000046400 r14:0xc0000061a0 r15:0x2031 rip:0x464770 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x125 fs_base:0xc000046490 gs_base:0x0
}
{
  real_time:15653.434366 global_time:429, event:`SYSCALL: pipe2' (state:EXITING_SYSCALL) tid:150135, ticks:1607
rax:0x0 rbx:0x0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x80800 rdi:0xc00006c7c0 rbp:0xc00006c7d0 rsp:0xc00006c7b0 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006c8e0 r13:0xc000046400 r14:0xc0000061a0 r15:0x2031 rip:0x464770 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x125 fs_base:0xc000046490 gs_base:0x0
  { tid:150135, addr:0xc00006c7c0, length:0x8 }
}
{
  real_time:15653.434380 global_time:430, event:`SYSCALL: epoll_ctl' (state:ENTERING_SYSCALL) tid:150135, ticks:1609
rax:0xffffffffffffffda rbx:0x5 rcx:0xffffffffffffffff rdx:0x6 rsi:0x1 rdi:0x5 rbp:0xc00006c7d0 rsp:0xc00006c788 r8:0x0 r9:0x0 r10:0xc00006c814 r11:0x246 r12:0xc00006c8e0 r13:0xc000046400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe9 fs_base:0xc000046490 gs_base:0x0
}
{
  real_time:15653.434391 global_time:431, event:`SYSCALL: epoll_ctl' (state:EXITING_SYSCALL) tid:150135, ticks:1609
rax:0x0 rbx:0x5 rcx:0xffffffffffffffff rdx:0x6 rsi:0x1 rdi:0x5 rbp:0xc00006c7d0 rsp:0xc00006c788 r8:0x0 r9:0x0 r10:0xc00006c814 r11:0x246 r12:0xc00006c8e0 r13:0xc000046400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe9 fs_base:0xc000046490 gs_base:0x0
}
{
  real_time:15653.434407 global_time:432, event:`SYSCALL: epoll_ctl' (state:ENTERING_SYSCALL) tid:150135, ticks:1717
rax:0xffffffffffffffda rbx:0x5 rcx:0xffffffffffffffff rdx:0x4 rsi:0x1 rdi:0x5 rbp:0xc00006c8e0 rsp:0xc00006c898 r8:0x0 r9:0x0 r10:0xc00006c90c r11:0x246 r12:0x0 r13:0xc000046400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe9 fs_base:0xc000046490 gs_base:0x0
}
{
  real_time:15653.434416 global_time:433, event:`SYSCALL: epoll_ctl' (state:EXITING_SYSCALL) tid:150135, ticks:1717
rax:0xffffffffffffffff rbx:0x5 rcx:0xffffffffffffffff rdx:0x4 rsi:0x1 rdi:0x5 rbp:0xc00006c8e0 rsp:0xc00006c898 r8:0x0 r9:0x0 r10:0xc00006c90c r11:0x246 r12:0x0 r13:0xc000046400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe9 fs_base:0xc000046490 gs_base:0x0
}
{
  real_time:15653.434429 global_time:434, event:`SYSCALL: fcntl' (state:ENTERING_SYSCALL) tid:150135, ticks:1777
rax:0xffffffffffffffda rbx:0x4 rcx:0xffffffffffffffff rdx:0x0 rsi:0x3 rdi:0x4 rbp:0xc00006c898 rsp:0xc00006c858 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x0 r13:0xc000046400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0xc000046490 gs_base:0x0
}
{
  real_time:15653.434438 global_time:435, event:`SYSCALL: fcntl' (state:EXITING_SYSCALL) tid:150135, ticks:1777
rax:0x8802 rbx:0x4 rcx:0xffffffffffffffff rdx:0x0 rsi:0x3 rdi:0x4 rbp:0xc00006c898 rsp:0xc00006c858 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x0 r13:0xc000046400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0xc000046490 gs_base:0x0
}
{
  real_time:15653.434451 global_time:436, event:`SYSCALL: fcntl' (state:ENTERING_SYSCALL) tid:150135, ticks:1824
rax:0xffffffffffffffda rbx:0x4 rcx:0xffffffffffffffff rdx:0x8002 rsi:0x4 rdi:0x4 rbp:0xc00006c898 rsp:0xc00006c858 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x0 r13:0xc000046400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0xc000046490 gs_base:0x0
}
{
  real_time:15653.434460 global_time:437, event:`SYSCALL: fcntl' (state:EXITING_SYSCALL) tid:150135, ticks:1824
rax:0x0 rbx:0x4 rcx:0xffffffffffffffff rdx:0x8002 rsi:0x4 rdi:0x4 rbp:0xc00006c898 rsp:0xc00006c858 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x0 r13:0xc000046400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0xc000046490 gs_base:0x0
}
{
  real_time:15653.434475 global_time:438, event:`SYSCALL: ftruncate' (state:ENTERING_SYSCALL) tid:150135, ticks:2018
rax:0xffffffffffffffda rbx:0x4 rcx:0xffffffffffffffff rdx:0x0 rsi:0x1000 rdi:0x4 rbp:0xc00006cab0 rsp:0xc00006ca70 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006ccf8 r13:0xc000046400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x4d fs_base:0xc000046490 gs_base:0x0
}
{
  real_time:15653.434492 global_time:439, event:`SYSCALL: ftruncate' (state:EXITING_SYSCALL) tid:150135, ticks:2018
rax:0x0 rbx:0x4 rcx:0xffffffffffffffff rdx:0x0 rsi:0x1000 rdi:0x4 rbp:0xc00006cab0 rsp:0xc00006ca70 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006ccf8 r13:0xc000046400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x4d fs_base:0xc000046490 gs_base:0x0
}
{
  real_time:15653.434507 global_time:440, event:`SYSCALL: fstat' (state:ENTERING_SYSCALL) tid:150135, ticks:2159
rax:0xffffffffffffffda rbx:0x4 rcx:0xffffffffffffffff rdx:0x0 rsi:0xc000072518 rdi:0x4 rbp:0xc00006c8c0 rsp:0xc00006c880 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006cb10 r13:0xc000046400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x5 fs_base:0xc000046490 gs_base:0x0
}
{
  real_time:15653.434518 global_time:441, event:`SYSCALL: fstat' (state:EXITING_SYSCALL) tid:150135, ticks:2159
rax:0x0 rbx:0x4 rcx:0xffffffffffffffff rdx:0x0 rsi:0xc000072518 rdi:0x4 rbp:0xc00006c8c0 rsp:0xc00006c880 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006cb10 r13:0xc000046400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x5 fs_base:0xc000046490 gs_base:0x0
  { tid:150135, addr:0xc000072518, length:0x90 }
}
{
  real_time:15653.434542 global_time:442, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:150135, ticks:2276
rax:0xffffffffffffffda rbx:0x0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1000 rdi:0x0 rbp:0xc00006c718 rsp:0xc00006c6d8 r8:0x4 r9:0x0 r10:0x1 r11:0x246 r12:0xc00006c8b0 r13:0xc000046400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0xc000046490 gs_base:0x0
}
{
  real_time:15653.434817 global_time:443, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:150135, ticks:2276
rax:0x7ff22325c000 rbx:0x0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1000 rdi:0x0 rbp:0xc00006c718 rsp:0xc00006c6d8 r8:0x4 r9:0x0 r10:0x1 r11:0x246 r12:0xc00006c8b0 r13:0xc000046400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0xc000046490 gs_base:0x0
  { map_file:"/tmp/1246591439", addr:0x7ff22325c000, length:0x1000, prot_flags:"rw-s", file_offset:0x0, device:2051, inode:275749, data_file:"", data_offset:0x0, file_size:0x1000 }
  { tid:150135, addr:0x7ff22325c000, length:0x1000, holes:[(nil)-0x1000] }
}
{
  real_time:15653.434855 global_time:444, event:`SYSCALL: munmap' (state:ENTERING_SYSCALL) tid:150135, ticks:2612
rax:0xffffffffffffffda rbx:0x7ff22325c000 rcx:0xffffffffffffffff rdx:0x0 rsi:0x1000 rdi:0x7ff22325c000 rbp:0xc00006c9d8 rsp:0xc00006c998 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006cb58 r13:0xc000046400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xb fs_base:0xc000046490 gs_base:0x0
}
[FATAL src/log.cc:444:emergency_debug()] (session doesn't look interactive, aborting emergency debugging)
=== Start rr backtrace:
rr(_ZN2rr13dump_rr_stackEv+0x41)[0x55a5ff013b61]
rr(_ZN2rr15notifying_abortEv+0xe)[0x55a5ff01509e]
rr(+0x1bdf0f)[0x55a5ff02af0f]
rr(+0xabe43)[0x55a5fef18e43]
rr(+0xac8cb)[0x55a5fef198cb]
rr(_ZN2rr18AutoRemoteSyscalls20check_syscall_resultElib+0x186)[0x55a5feec8896]
rr(_ZN2rr18AutoRemoteSyscalls18finish_direct_mmapENS_10remote_ptrIvEEmiiRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEilR4statRS8_+0x1fd)[0x55a5feec98bd]
rr(+0x139e71)[0x55a5fefa6e71]
rr(+0x13b377)[0x55a5fefa8377]
rr(+0x13fe37)[0x55a5feface37]
rr(_ZN2rr13ReplaySession28setup_replay_one_trace_frameEPNS_10ReplayTaskE+0x7b7)[0x55a5fefa4367]
rr(_ZN2rr13ReplaySession11replay_stepERKNS0_15StepConstraintsE+0x9c)[0x55a5fefa454c]
rr(_ZN2rr14ReplayTimeline19replay_step_forwardENS_10RunCommandE+0xc4)[0x55a5fefbdd94]
rr(_ZN2rr9GdbServer14debug_one_stepERNS_10GdbRequestE+0x3c5)[0x55a5fef05e25]
rr(_ZN2rr9GdbServer12serve_replayERKNS0_15ConnectionFlagsE+0x69b)[0x55a5fef0726b]
rr(+0x12c61a)[0x55a5fef9961a]
rr(_ZN2rr13ReplayCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x63f)[0x55a5fef9ae6f]
rr(main+0x1ad)[0x55a5feeb383d]
/usr/lib64/libc.so.6(+0x238ca)[0x7fa43b73f8ca]
/usr/lib64/libc.so.6(__libc_start_main+0x85)[0x7fa43b73f985]
rr(_start+0x21)[0x55a5feeb3a11]
=== End rr backtrace
2023-10-24T14:49:15-05:00 error layer=dap runtime error: EOF
Unable to retrieve goroutines: write tcp 127.0.0.1:42628->127.0.0.1:19064: write: broken pipe
Detaching and terminating target process
2023-10-24T14:51:56-05:00 error layer=dap detach returned error: could not detach process 0: write tcp 127.0.0.1:42628->127.0.0.1:19064: write: broken pipe
dlv dap (149952) exited with code: 0
  • Complete example, including launch config

  • My kernel config

  • uname -a: Linux MyPC 6.1.38-gentoo #1 SMP PREEMPT_DYNAMIC Thu Aug 3 12:07:10 CDT 2023 x86_64 Intel(R) Core(TM) i9-10850K CPU @ 3.60GHz GenuineIntel GNU/Linux

  • rr --version: rr version 5.6.0

  • go version: go version go1.20.2 linux/amd64

  • dlv version: Version: 1.21.1; Build: $Id: a358c02f24aa7047ecc562b0587dc2d08330b2cf $

  • VSCode version:

    Version: 1.82.2
    Commit: abd2f3db4bdb28f9e95536dfa84d8479f1eb312d
    Date: 2023-09-14T05:51:20.981Z
    Electron: 25.8.1
    ElectronBuildId: 23779380
    Chromium: 114.0.5735.289
    Node.js: 18.15.0
    V8: 11.4.183.29-electron.0
    OS: Linux x64 6.1.38-gentoo

@rocallahan
Copy link
Collaborator

You might want to try with the latest rr version.

Try passing -F to rr and then when rr is paused for the emergency debugger, check if the referenced proc file (/proc/150136/fd/22 in your case) actually exists? And what its permissions are?

Also does recording and replaying simple stuff (e.g. rr record ls; rr replay -a) work?

@firelizzard18
Copy link
Author

It worked fine with func main() { fmt.Println("Hello world") }. It also worked on a far more complex test - I refactored my code such that I can pass a build flag (equivalent to -DNO_MMAP) that will prevent the mmap call from being compiled in. Using mmap makes that code much faster but it's not necessary, so I disabled it and ran the test without it.

@rocallahan
Copy link
Collaborator

This file seems to be malformed, gunzip barfs.

@firelizzard18
Copy link
Author

This file seems to be malformed, gunzip barfs.

Oops, looks like it's actually XZ encrypted not gzip, I must have used the wrong flag:

❯ file example.tar.gz 
example.tar.gz: XZ compressed data, checksum CRC64

tar xf example.tar.gz works fine on my system, replacing .gz with .xz might help.

The part that actually mattered was the VSCode config which apparently did not get included. The tarball just includes the source, go.mod, and go.sum. I'm attaching a new tarball that is actually gzipped and includes .vscode. I'll also include .vscode/launch.json as its own file, for redundancy.

@rocallahan
Copy link
Collaborator

Works for me with gdb.

[roc@localhost y]$ go build -gcflags '-N -l'
[roc@localhost y]$ rr record foo
rr: Saving execution to trace directory `/home/roc/.local/share/rr/foo-1'.
[roc@localhost y]$ rr replay
GNU gdb (GDB) Fedora Linux 13.2-5.fc38
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Warning: /usr/local/lib/rustlib/etc: No such file or directory.
Reading symbols from /home/roc/.local/share/rr/foo-1/mmap_clone_3_foo...
Loading Go Runtime support.
Really redefine built-in command "restart"? (y or n) [answered Y; input not from terminal]
Really redefine built-in command "jump"? (y or n) [answered Y; input not from terminal]
Remote debugging using 127.0.0.1:18056
BFD: warning: system-supplied DSO at 0x6fffd000 has a section extending past end of file
_rt0_amd64_linux () at /usr/lib/golang/src/runtime/rt0_linux_amd64.s:8
8		JMP	_rt0_amd64(SB)
(rr) break 'github.com/edsrzf/mmap-go.Map'
Breakpoint 1 at 0x473746: file /home/roc/go/pkg/mod/github.com/edsrzf/[email protected]/mmap.go, line 48.
(rr) c
Continuing.
[New Thread 1787520.1787521]
[New Thread 1787520.1787522]

Thread 1 hit Breakpoint 1, github.com/edsrzf/mmap-go.Map (f=0xc000012030, prot=2, flags=0, ~r0=..., ~r1=...)
    at /home/roc/go/pkg/mod/github.com/edsrzf/[email protected]/mmap.go:48
48	func Map(f *os.File, prot, flags int) (MMap, error) {

@rocallahan
Copy link
Collaborator

Can you run dlv from the command line to reproduce your bug? I have no idea how to set up VSCode to run dlv to debug Go programs and it's much easier to reproduce and debug stuff running from the command line.

@firelizzard18
Copy link
Author

Sure, I’ll ping you when I’ve done that

@firelizzard18
Copy link
Author

@rocallahan When I unpack the tarball into a directory and run the following in that directory:

$ dlv debug --backend rr . # Start debugging with delve
(dlv) break main.main      # Put a breakpoint at the beginning of main
(dlv) continue             # Start the program
(dlv) next                 # Step (repeat 5 more times)
It crashes:
[FATAL src/AutoRemoteSyscalls.cc:734:check_syscall_result()] 
 (task 238653 (rec:238612) at time 455)
 -> Assertion `false' failed to hold. Syscall openat failed with errno EACCES opening /proc/238613/fd/21
Tail of trace dump:
{
  real_time:24338.509139 global_time:435, event:`SYSCALL: fcntl' (state:EXITING_SYSCALL) tid:238612, ticks:1534
rax:0x8002 rbx:0x4 rcx:0xffffffffffffffff rdx:0x0 rsi:0x3 rdi:0x4 rbp:0xc00006e898 rsp:0xc00006e858 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006e930 r13:0xc000048400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0xc000048490 gs_base:0x0
}
{
  real_time:24338.509162 global_time:436, event:`SYSCALL: fcntl' (state:ENTERING_SYSCALL) tid:238612, ticks:1576
rax:0xffffffffffffffda rbx:0x4 rcx:0xffffffffffffffff rdx:0x8802 rsi:0x4 rdi:0x4 rbp:0xc00006e898 rsp:0xc00006e858 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006e930 r13:0xc000048400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0xc000048490 gs_base:0x0
}
{
  real_time:24338.509182 global_time:437, event:`SYSCALL: fcntl' (state:EXITING_SYSCALL) tid:238612, ticks:1576
rax:0x0 rbx:0x4 rcx:0xffffffffffffffff rdx:0x8802 rsi:0x4 rdi:0x4 rbp:0xc00006e898 rsp:0xc00006e858 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006e930 r13:0xc000048400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0xc000048490 gs_base:0x0
}
{
  real_time:24338.509204 global_time:438, event:`SYSCALL: epoll_create1' (state:ENTERING_SYSCALL) tid:238612, ticks:1629
rax:0xffffffffffffffda rbx:0x80000 rcx:0xffffffffffffffff rdx:0x0 rsi:0x0 rdi:0x80000 rbp:0xc00006e7d0 rsp:0xc00006e790 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006e8e0 r13:0xc000048400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x123 fs_base:0xc000048490 gs_base:0x0
}
{
  real_time:24338.509222 global_time:439, event:`SYSCALL: epoll_create1' (state:EXITING_SYSCALL) tid:238612, ticks:1629
rax:0x5 rbx:0x80000 rcx:0xffffffffffffffff rdx:0x0 rsi:0x0 rdi:0x80000 rbp:0xc00006e7d0 rsp:0xc00006e790 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006e8e0 r13:0xc000048400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x123 fs_base:0xc000048490 gs_base:0x0
}
{
  real_time:24338.509244 global_time:440, event:`SYSCALL: pipe2' (state:ENTERING_SYSCALL) tid:238612, ticks:1632
rax:0xffffffffffffffda rbx:0x0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x80800 rdi:0xc00006e7c0 rbp:0xc00006e7d0 rsp:0xc00006e7b0 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006e8e0 r13:0xc000048400 r14:0xc0000061a0 r15:0x2031 rip:0x464770 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x125 fs_base:0xc000048490 gs_base:0x0
}
{
  real_time:24338.509265 global_time:441, event:`SYSCALL: pipe2' (state:EXITING_SYSCALL) tid:238612, ticks:1632
rax:0x0 rbx:0x0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x80800 rdi:0xc00006e7c0 rbp:0xc00006e7d0 rsp:0xc00006e7b0 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006e8e0 r13:0xc000048400 r14:0xc0000061a0 r15:0x2031 rip:0x464770 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x125 fs_base:0xc000048490 gs_base:0x0
  { tid:238612, addr:0xc00006e7c0, length:0x8 }
}
{
  real_time:24338.509287 global_time:442, event:`SYSCALL: epoll_ctl' (state:ENTERING_SYSCALL) tid:238612, ticks:1634
rax:0xffffffffffffffda rbx:0x5 rcx:0xffffffffffffffff rdx:0x6 rsi:0x1 rdi:0x5 rbp:0xc00006e7d0 rsp:0xc00006e788 r8:0x0 r9:0x0 r10:0xc00006e814 r11:0x246 r12:0xc00006e8e0 r13:0xc000048400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe9 fs_base:0xc000048490 gs_base:0x0
}
{
  real_time:24338.509304 global_time:443, event:`SYSCALL: epoll_ctl' (state:EXITING_SYSCALL) tid:238612, ticks:1634
rax:0x0 rbx:0x5 rcx:0xffffffffffffffff rdx:0x6 rsi:0x1 rdi:0x5 rbp:0xc00006e7d0 rsp:0xc00006e788 r8:0x0 r9:0x0 r10:0xc00006e814 r11:0x246 r12:0xc00006e8e0 r13:0xc000048400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe9 fs_base:0xc000048490 gs_base:0x0
}
{
  real_time:24338.509330 global_time:444, event:`SYSCALL: epoll_ctl' (state:ENTERING_SYSCALL) tid:238612, ticks:1742
rax:0xffffffffffffffda rbx:0x5 rcx:0xffffffffffffffff rdx:0x4 rsi:0x1 rdi:0x5 rbp:0xc00006e8e0 rsp:0xc00006e898 r8:0x0 r9:0x0 r10:0xc00006e90c r11:0x246 r12:0x0 r13:0xc000048400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe9 fs_base:0xc000048490 gs_base:0x0
}
{
  real_time:24338.509344 global_time:445, event:`SYSCALL: epoll_ctl' (state:EXITING_SYSCALL) tid:238612, ticks:1742
rax:0xffffffffffffffff rbx:0x5 rcx:0xffffffffffffffff rdx:0x4 rsi:0x1 rdi:0x5 rbp:0xc00006e8e0 rsp:0xc00006e898 r8:0x0 r9:0x0 r10:0xc00006e90c r11:0x246 r12:0x0 r13:0xc000048400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe9 fs_base:0xc000048490 gs_base:0x0
}
{
  real_time:24338.509366 global_time:446, event:`SYSCALL: fcntl' (state:ENTERING_SYSCALL) tid:238612, ticks:1802
rax:0xffffffffffffffda rbx:0x4 rcx:0xffffffffffffffff rdx:0x0 rsi:0x3 rdi:0x4 rbp:0xc00006e898 rsp:0xc00006e858 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x0 r13:0xc000048400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0xc000048490 gs_base:0x0
}
{
  real_time:24338.509381 global_time:447, event:`SYSCALL: fcntl' (state:EXITING_SYSCALL) tid:238612, ticks:1802
rax:0x8802 rbx:0x4 rcx:0xffffffffffffffff rdx:0x0 rsi:0x3 rdi:0x4 rbp:0xc00006e898 rsp:0xc00006e858 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x0 r13:0xc000048400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0xc000048490 gs_base:0x0
}
{
  real_time:24338.509401 global_time:448, event:`SYSCALL: fcntl' (state:ENTERING_SYSCALL) tid:238612, ticks:1844
rax:0xffffffffffffffda rbx:0x4 rcx:0xffffffffffffffff rdx:0x8002 rsi:0x4 rdi:0x4 rbp:0xc00006e898 rsp:0xc00006e858 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x0 r13:0xc000048400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0xc000048490 gs_base:0x0
}
{
  real_time:24338.509416 global_time:449, event:`SYSCALL: fcntl' (state:EXITING_SYSCALL) tid:238612, ticks:1844
rax:0x0 rbx:0x4 rcx:0xffffffffffffffff rdx:0x8002 rsi:0x4 rdi:0x4 rbp:0xc00006e898 rsp:0xc00006e858 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x0 r13:0xc000048400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0xc000048490 gs_base:0x0
}
{
  real_time:24338.509440 global_time:450, event:`SYSCALL: ftruncate' (state:ENTERING_SYSCALL) tid:238612, ticks:2035
rax:0xffffffffffffffda rbx:0x4 rcx:0xffffffffffffffff rdx:0x0 rsi:0x1000 rdi:0x4 rbp:0xc00006eab0 rsp:0xc00006ea70 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006ecf8 r13:0xc000048400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x4d fs_base:0xc000048490 gs_base:0x0
}
{
  real_time:24338.509461 global_time:451, event:`SYSCALL: ftruncate' (state:EXITING_SYSCALL) tid:238612, ticks:2035
rax:0x0 rbx:0x4 rcx:0xffffffffffffffff rdx:0x0 rsi:0x1000 rdi:0x4 rbp:0xc00006eab0 rsp:0xc00006ea70 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006ecf8 r13:0xc000048400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x4d fs_base:0xc000048490 gs_base:0x0
}
{
  real_time:24338.509485 global_time:452, event:`SYSCALL: fstat' (state:ENTERING_SYSCALL) tid:238612, ticks:2184
rax:0xffffffffffffffda rbx:0x4 rcx:0xffffffffffffffff rdx:0x0 rsi:0xc000074448 rdi:0x4 rbp:0xc00006e8c0 rsp:0xc00006e880 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006eb10 r13:0xc000048400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x5 fs_base:0xc000048490 gs_base:0x0
}
{
  real_time:24338.509501 global_time:453, event:`SYSCALL: fstat' (state:EXITING_SYSCALL) tid:238612, ticks:2184
rax:0x0 rbx:0x4 rcx:0xffffffffffffffff rdx:0x0 rsi:0xc000074448 rdi:0x4 rbp:0xc00006e8c0 rsp:0xc00006e880 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006eb10 r13:0xc000048400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x5 fs_base:0xc000048490 gs_base:0x0
  { tid:238612, addr:0xc000074448, length:0x90 }
}
{
  real_time:24338.509527 global_time:454, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:238612, ticks:2298
rax:0xffffffffffffffda rbx:0x0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1000 rdi:0x0 rbp:0xc00006e718 rsp:0xc00006e6d8 r8:0x4 r9:0x0 r10:0x1 r11:0x246 r12:0xc00006e8b0 r13:0xc000048400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0xc000048490 gs_base:0x0
}
{
  real_time:24338.509666 global_time:455, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:238612, ticks:2298
rax:0x7fbe47229000 rbx:0x0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1000 rdi:0x0 rbp:0xc00006e718 rsp:0xc00006e6d8 r8:0x4 r9:0x0 r10:0x1 r11:0x246 r12:0xc00006e8b0 r13:0xc000048400 r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0xc000048490 gs_base:0x0
  { map_file:"/tmp/911353397", addr:0x7fbe47229000, length:0x1000, prot_flags:"rw-s", file_offset:0x0, device:2051, inode:280516, data_file:"", data_offset:0x0, file_size:0x1000 }
  { tid:238612, addr:0x7fbe47229000, length:0x1000, holes:[(nil)-0x1000] }
}
{
  real_time:24338.509708 global_time:456, event:`SYSCALL: munmap' (state:ENTERING_SYSCALL) tid:238612, ticks:2819
rax:0xffffffffffffffda rbx:0x7fbe47229000 rcx:0xffffffffffffffff rdx:0x0 rsi:0x1000 rdi:0x7fbe47229000 rbp:0xc00006e9d8 rsp:0xc00006e998 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006eb58 r13:0xffffffffffffffff r14:0xc0000061a0 r15:0x2031 rip:0x4034ee eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xb fs_base:0xc000048490 gs_base:0x0
}
[FATAL src/log.cc:444:emergency_debug()] (session doesn't look interactive, aborting emergency debugging)
=== Start rr backtrace:
rr(_ZN2rr13dump_rr_stackEv+0x41)[0x55eca82f3b61]
rr(_ZN2rr15notifying_abortEv+0xe)[0x55eca82f509e]
rr(+0x1bdf0f)[0x55eca830af0f]
rr(+0xabe43)[0x55eca81f8e43]
rr(+0xac8cb)[0x55eca81f98cb]
rr(_ZN2rr18AutoRemoteSyscalls20check_syscall_resultElib+0x186)[0x55eca81a8896]
rr(_ZN2rr18AutoRemoteSyscalls18finish_direct_mmapENS_10remote_ptrIvEEmiiRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEilR4statRS8_+0x1fd)[0x55eca81a98bd]
rr(+0x139e71)[0x55eca8286e71]
rr(+0x13b377)[0x55eca8288377]
rr(+0x13fe37)[0x55eca828ce37]
rr(_ZN2rr13ReplaySession28setup_replay_one_trace_frameEPNS_10ReplayTaskE+0x7b7)[0x55eca8284367]
rr(_ZN2rr13ReplaySession11replay_stepERKNS0_15StepConstraintsE+0x9c)[0x55eca828454c]
rr(_ZN2rr14ReplayTimeline19replay_step_forwardENS_10RunCommandE+0xc4)[0x55eca829dd94]
rr(_ZN2rr9GdbServer14debug_one_stepERNS_10GdbRequestE+0x3c5)[0x55eca81e5e25]
rr(_ZN2rr9GdbServer12serve_replayERKNS0_15ConnectionFlagsE+0x69b)[0x55eca81e726b]
rr(+0x12c61a)[0x55eca827961a]
rr(_ZN2rr13ReplayCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x63f)[0x55eca827ae6f]
rr(main+0x1ad)[0x55eca819383d]
/usr/lib64/libc.so.6(+0x238ca)[0x7f7d448ee8ca]
/usr/lib64/libc.so.6(__libc_start_main+0x85)[0x7f7d448ee985]
rr(_start+0x21)[0x55eca8193a11]
=== End rr backtrace
> main.main() ./main.go:14 (PC: 0x4874fa)
Command failed: EOF

I get the same result if I:

$ go build .
$ rr record foo
$ dlv replay /home/firelizzard/.local/share/rr/foo-0
(dlv) break main.main
(dlv) continue
(dlv) next # 6 times

@firelizzard18
Copy link
Author

Works for me with gdb.

@rocallahan Did you step past the mmap syscall? When I execute the same steps as you, using the same gcflags, rr record, and rr replay (gdb), I still get the assertion fail if I step twice:

log
$ go build -gcflags '-N -l'
$ rr record foo
rr: Saving execution to trace directory `/home/firelizzard/.local/share/rr/foo-1'.
$ rr replay
GNU gdb (Gentoo 13.2 vanilla) 13.2
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
    .

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /home/firelizzard/.local/share/rr/foo-1/mmap_copy_3_foo...
warning: File "/usr/lib/go/src/runtime/runtime-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
add-auto-load-safe-path /usr/lib/go/src/runtime/runtime-gdb.py
line to your configuration file "/home/firelizzard/.config/gdb/gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/home/firelizzard/.config/gdb/gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
info "(gdb)Auto-loading safe path"
Really redefine built-in command "restart"? (y or n) [answered Y; input not from terminal]
Really redefine built-in command "jump"? (y or n) [answered Y; input not from terminal]
Remote debugging using 127.0.0.1:44532
_rt0_amd64_linux () at /usr/lib/go/src/runtime/rt0_linux_amd64.s:8
8 JMP _rt0_amd64(SB)
(rr) break 'github.com/edsrzf/mmap-go.Map'
Breakpoint 1 at 0x46d70a: file /home/firelizzard/go/pkg/mod/github.com/edsrzf/[email protected]/mmap.go, line 48.
(rr) c
Continuing.
[New Thread 241120.241121]
[New Thread 241120.241122]

Thread 1 hit Breakpoint 1, github.com/edsrzf/mmap-go.Map (f=0xc000030028, prot=2, flags=0, ~r0=..., ~r1=...)
at /home/firelizzard/go/pkg/mod/github.com/edsrzf/[email protected]/mmap.go:48
48 func Map(f *os.File, prot, flags int) (MMap, error) {
(rr) n
49 return MapRegion(f, -1, prot, flags, 0)
(rr) n
[FATAL src/AutoRemoteSyscalls.cc:734:check_syscall_result()]
(task 241141 (rec:241120) at time 437)
-> Assertion false' failed to hold. Syscall openat failed with errno EACCES opening /proc/241140/fd/21 Tail of trace dump: { real_time:24834.462592 global_time:417, event:SYSCALL: fcntl' (state:EXITING_SYSCALL) tid:241120, ticks:59971
rax:0x8002 rbx:0x3 rcx:0xffffffffffffffff rdx:0x0 rsi:0x3 rdi:0x3 rbp:0xc000072b08 rsp:0xc000072ac8 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x1000000000000 r13:0x28828828878f r14:0xc0000061a0 r15:0x4 rip:0x40286e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0x4fd110 gs_base:0x0
}
{
real_time:24834.462605 global_time:418, event:SYSCALL: fcntl' (state:ENTERING_SYSCALL) tid:241120, ticks:60012 rax:0xffffffffffffffda rbx:0x3 rcx:0xffffffffffffffff rdx:0x8802 rsi:0x4 rdi:0x3 rbp:0xc000072b08 rsp:0xc000072ac8 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x1000000000000 r13:0x28828828878f r14:0xc0000061a0 r15:0x4 rip:0x40286e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0x4fd110 gs_base:0x0 } { real_time:24834.462613 global_time:419, event:SYSCALL: fcntl' (state:EXITING_SYSCALL) tid:241120, ticks:60012
rax:0x0 rbx:0x3 rcx:0xffffffffffffffff rdx:0x8802 rsi:0x4 rdi:0x3 rbp:0xc000072b08 rsp:0xc000072ac8 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x1000000000000 r13:0x28828828878f r14:0xc0000061a0 r15:0x4 rip:0x40286e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0x4fd110 gs_base:0x0
}
{
real_time:24834.462626 global_time:420, event:SYSCALL: epoll_create1' (state:ENTERING_SYSCALL) tid:241120, ticks:60052 rax:0xffffffffffffffda rbx:0x80000 rcx:0xffffffffffffffff rdx:0x0 rsi:0x0 rdi:0x80000 rbp:0xc000072b18 rsp:0xc000072aa0 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x1000000000000 r13:0x28828828878f r14:0xc0000061a0 r15:0x4 rip:0x40286e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x123 fs_base:0x4fd110 gs_base:0x0 } { real_time:24834.462637 global_time:421, event:SYSCALL: epoll_create1' (state:EXITING_SYSCALL) tid:241120, ticks:60052
rax:0x4 rbx:0x80000 rcx:0xffffffffffffffff rdx:0x0 rsi:0x0 rdi:0x80000 rbp:0xc000072b18 rsp:0xc000072aa0 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x1000000000000 r13:0x28828828878f r14:0xc0000061a0 r15:0x4 rip:0x40286e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x123 fs_base:0x4fd110 gs_base:0x0
}
{
real_time:24834.462650 global_time:422, event:SYSCALL: pipe2' (state:ENTERING_SYSCALL) tid:241120, ticks:60054 rax:0xffffffffffffffda rbx:0x0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x80800 rdi:0xc000072ab0 rbp:0xc000072b18 rsp:0xc000072aa0 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x1000000000000 r13:0x28828828878f r14:0xc0000061a0 r15:0x4 rip:0x45bc30 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x125 fs_base:0x4fd110 gs_base:0x0 } { real_time:24834.462666 global_time:423, event:SYSCALL: pipe2' (state:EXITING_SYSCALL) tid:241120, ticks:60054
rax:0x0 rbx:0x0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x80800 rdi:0xc000072ab0 rbp:0xc000072b18 rsp:0xc000072aa0 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x1000000000000 r13:0x28828828878f r14:0xc0000061a0 r15:0x4 rip:0x45bc30 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x125 fs_base:0x4fd110 gs_base:0x0
{ tid:241120, addr:0xc000072ab0, length:0x8 }
}
{
real_time:24834.462679 global_time:424, event:SYSCALL: epoll_ctl' (state:ENTERING_SYSCALL) tid:241120, ticks:60055 rax:0xffffffffffffffda rbx:0x4 rcx:0xffffffffffffffff rdx:0x5 rsi:0x1 rdi:0x4 rbp:0xc000072b18 rsp:0xc000072aa0 r8:0x0 r9:0x0 r10:0xc000072b04 r11:0x246 r12:0x1000000000000 r13:0x28828828878f r14:0xc0000061a0 r15:0x4 rip:0x40286e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe9 fs_base:0x4fd110 gs_base:0x0 } { real_time:24834.462690 global_time:425, event:SYSCALL: epoll_ctl' (state:EXITING_SYSCALL) tid:241120, ticks:60055
rax:0x0 rbx:0x4 rcx:0xffffffffffffffff rdx:0x5 rsi:0x1 rdi:0x4 rbp:0xc000072b18 rsp:0xc000072aa0 r8:0x0 r9:0x0 r10:0xc000072b04 r11:0x246 r12:0x1000000000000 r13:0x28828828878f r14:0xc0000061a0 r15:0x4 rip:0x40286e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe9 fs_base:0x4fd110 gs_base:0x0
}
{
real_time:24834.462705 global_time:426, event:SYSCALL: epoll_ctl' (state:ENTERING_SYSCALL) tid:241120, ticks:60132 rax:0xffffffffffffffda rbx:0x4 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1 rdi:0x4 rbp:0xc000072b60 rsp:0xc000072b08 r8:0x0 r9:0x0 r10:0xc000072b4c r11:0x246 r12:0x0 r13:0x28828828878f r14:0xc0000061a0 r15:0x4 rip:0x40286e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe9 fs_base:0x4fd110 gs_base:0x0 } { real_time:24834.462714 global_time:427, event:SYSCALL: epoll_ctl' (state:EXITING_SYSCALL) tid:241120, ticks:60132
rax:0xffffffffffffffff rbx:0x4 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1 rdi:0x4 rbp:0xc000072b60 rsp:0xc000072b08 r8:0x0 r9:0x0 r10:0xc000072b4c r11:0x246 r12:0x0 r13:0x28828828878f r14:0xc0000061a0 r15:0x4 rip:0x40286e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe9 fs_base:0x4fd110 gs_base:0x0
}
{
real_time:24834.462726 global_time:428, event:SYSCALL: fcntl' (state:ENTERING_SYSCALL) tid:241120, ticks:60181 rax:0xffffffffffffffda rbx:0x3 rcx:0xffffffffffffffff rdx:0x0 rsi:0x3 rdi:0x3 rbp:0xc000072b08 rsp:0xc000072ac8 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x0 r13:0x28828828878f r14:0xc0000061a0 r15:0x4 rip:0x40286e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0x4fd110 gs_base:0x0 } { real_time:24834.462734 global_time:429, event:SYSCALL: fcntl' (state:EXITING_SYSCALL) tid:241120, ticks:60181
rax:0x8802 rbx:0x3 rcx:0xffffffffffffffff rdx:0x0 rsi:0x3 rdi:0x3 rbp:0xc000072b08 rsp:0xc000072ac8 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x0 r13:0x28828828878f r14:0xc0000061a0 r15:0x4 rip:0x40286e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0x4fd110 gs_base:0x0
}
{
real_time:24834.462747 global_time:430, event:SYSCALL: fcntl' (state:ENTERING_SYSCALL) tid:241120, ticks:60222 rax:0xffffffffffffffda rbx:0x3 rcx:0xffffffffffffffff rdx:0x8002 rsi:0x4 rdi:0x3 rbp:0xc000072b08 rsp:0xc000072ac8 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x0 r13:0x28828828878f r14:0xc0000061a0 r15:0x4 rip:0x40286e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0x4fd110 gs_base:0x0 } { real_time:24834.462755 global_time:431, event:SYSCALL: fcntl' (state:EXITING_SYSCALL) tid:241120, ticks:60222
rax:0x0 rbx:0x3 rcx:0xffffffffffffffff rdx:0x8002 rsi:0x4 rdi:0x3 rbp:0xc000072b08 rsp:0xc000072ac8 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x0 r13:0x28828828878f r14:0xc0000061a0 r15:0x4 rip:0x40286e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0x4fd110 gs_base:0x0
}
{
real_time:24834.462770 global_time:432, event:SYSCALL: ftruncate' (state:ENTERING_SYSCALL) tid:241120, ticks:60379 rax:0xffffffffffffffda rbx:0x3 rcx:0xffffffffffffffff rdx:0x0 rsi:0x1000 rdi:0x3 rbp:0xc000072c48 rsp:0xc000072c08 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc000072b98 r13:0x28828828878f r14:0xc0000061a0 r15:0x4 rip:0x40286e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x4d fs_base:0x4fd110 gs_base:0x0 } { real_time:24834.462790 global_time:433, event:SYSCALL: ftruncate' (state:EXITING_SYSCALL) tid:241120, ticks:60379
rax:0x0 rbx:0x3 rcx:0xffffffffffffffff rdx:0x0 rsi:0x1000 rdi:0x3 rbp:0xc000072c48 rsp:0xc000072c08 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc000072b98 r13:0x28828828878f r14:0xc0000061a0 r15:0x4 rip:0x40286e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x4d fs_base:0x4fd110 gs_base:0x0
}
{
real_time:24834.462805 global_time:434, event:SYSCALL: fstat' (state:ENTERING_SYSCALL) tid:241120, ticks:60498 rax:0xffffffffffffffda rbx:0x3 rcx:0xffffffffffffffff rdx:0x0 rsi:0xc00006c378 rdi:0x3 rbp:0xc000072bb8 rsp:0xc000072b78 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x10000000000 r13:0x802aaaa00a r14:0xc0000061a0 r15:0x4 rip:0x40286e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x5 fs_base:0x4fd110 gs_base:0x0 } { real_time:24834.462815 global_time:435, event:SYSCALL: fstat' (state:EXITING_SYSCALL) tid:241120, ticks:60498
rax:0x0 rbx:0x3 rcx:0xffffffffffffffff rdx:0x0 rsi:0xc00006c378 rdi:0x3 rbp:0xc000072bb8 rsp:0xc000072b78 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x10000000000 r13:0x802aaaa00a r14:0xc0000061a0 r15:0x4 rip:0x40286e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x5 fs_base:0x4fd110 gs_base:0x0
{ tid:241120, addr:0xc00006c378, length:0x90 }
}
{
real_time:24834.462830 global_time:436, event:SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:241120, ticks:60604 rax:0xffffffffffffffda rbx:0x0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1000 rdi:0x0 rbp:0xc000072af0 rsp:0xc000072ab0 r8:0x3 r9:0x0 r10:0x1 r11:0x246 r12:0x0 r13:0x802aaaa00a r14:0xc0000061a0 r15:0x4 rip:0x40286e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x4fd110 gs_base:0x0 } { real_time:24834.463030 global_time:437, event:SYSCALL: mmap' (state:EXITING_SYSCALL) tid:241120, ticks:60604
rax:0x7f5f0abae000 rbx:0x0 rcx:0xffffffffffffffff rdx:0x3 rsi:0x1000 rdi:0x0 rbp:0xc000072af0 rsp:0xc000072ab0 r8:0x3 r9:0x0 r10:0x1 r11:0x246 r12:0x0 r13:0x802aaaa00a r14:0xc0000061a0 r15:0x4 rip:0x40286e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0x4fd110 gs_base:0x0
{ map_file:"/tmp/2978062711", addr:0x7f5f0abae000, length:0x1000, prot_flags:"rw-s", file_offset:0x0, device:2051, inode:280518, data_file:"", data_offset:0x0, file_size:0x1000 }
{ tid:241120, addr:0x7f5f0abae000, length:0x1000, holes:[(nil)-0x1000] }
}
{
real_time:24834.463080 global_time:438, event:`SYSCALL: munmap' (state:ENTERING_SYSCALL) tid:241120, ticks:61047
rax:0xffffffffffffffda rbx:0x7f5f0abae000 rcx:0xffffffffffffffff rdx:0x0 rsi:0x1000 rdi:0x7f5f0abae000 rbp:0xc000072bc0 rsp:0xc000072b80 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x0 r13:0x0 r14:0xc0000061a0 r15:0x3 rip:0x40286e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xb fs_base:0x4fd110 gs_base:0x0
}
=== Start rr backtrace:
rr(_ZN2rr13dump_rr_stackEv+0x41)[0x565203b14b61]
rr(_ZN2rr9GdbServer15emergency_debugEPNS_4TaskE+0x158)[0x565203a0a538]
rr(+0xabc89)[0x565203a19c89]
rr(+0xac8cb)[0x565203a1a8cb]
rr(_ZN2rr18AutoRemoteSyscalls20check_syscall_resultElib+0x186)[0x5652039c9896]
rr(ZN2rr18AutoRemoteSyscalls18finish_direct_mmapENS_10remote_ptrIvEEmiiRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEilR4statRS8+0x1fd)[0x5652039ca8bd]
rr(+0x139e71)[0x565203aa7e71]
rr(+0x13b377)[0x565203aa9377]
rr(+0x13fe37)[0x565203aade37]
rr(_ZN2rr13ReplaySession28setup_replay_one_trace_frameEPNS_10ReplayTaskE+0x7b7)[0x565203aa5367]
rr(_ZN2rr13ReplaySession11replay_stepERKNS0_15StepConstraintsE+0x9c)[0x565203aa554c]
rr(_ZN2rr14ReplayTimeline19replay_step_forwardENS_10RunCommandE+0xc4)[0x565203abed94]
rr(_ZN2rr9GdbServer14debug_one_stepERNS_10GdbRequestE+0x3c5)[0x565203a06e25]
rr(_ZN2rr9GdbServer12serve_replayERKNS0_15ConnectionFlagsE+0x69b)[0x565203a0826b]
rr(+0x12d411)[0x565203a9b411]
rr(_ZN2rr13ReplayCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x63f)[0x565203a9be6f]
rr(main+0x1ad)[0x5652039b483d]
/usr/lib64/libc.so.6(+0x238ca)[0x7f5717cc38ca]
/usr/lib64/libc.so.6(__libc_start_main+0x85)[0x7f5717cc3985]
rr(_start+0x21)[0x5652039b4a11]
=== End rr backtrace
Launch gdb with
gdb '-l' '10000' '-ex' 'set sysroot /' '-ex' 'target extended-remote 127.0.0.1:44533' /home/firelizzard/.local/share/rr/foo-1/mmap_copy_3_foo

@rocallahan
Copy link
Collaborator

That's no problem here.

Thread 1 hit Breakpoint 1, github.com/edsrzf/mmap-go.Map (f=0xc000012030, prot=2, flags=0, ~r0=..., ~r1=...)
    at /home/roc/go/pkg/mod/github.com/edsrzf/[email protected]/mmap.go:48
48	func Map(f *os.File, prot, flags int) (MMap, error) {
(rr) stepi
0x000000000048ccd6	48	func Map(f *os.File, prot, flags int) (MMap, error) {
(rr) 
0x000000000048ccde	48	func Map(f *os.File, prot, flags int) (MMap, error) {
(rr) 
0x000000000048cce6	48	func Map(f *os.File, prot, flags int) (MMap, error) {
(rr) 
0x000000000048ccee	48	func Map(f *os.File, prot, flags int) (MMap, error) {
(rr) 
0x000000000048ccf6	48	func Map(f *os.File, prot, flags int) (MMap, error) {
(rr) 
0x000000000048ccfe	48	func Map(f *os.File, prot, flags int) (MMap, error) {
(rr) 
0x000000000048cd07	48	func Map(f *os.File, prot, flags int) (MMap, error) {
(rr) 
0x000000000048cd0d	48	func Map(f *os.File, prot, flags int) (MMap, error) {
(rr) 
49		return MapRegion(f, -1, prot, flags, 0)
(rr) 
0x000000000048cd1c	49		return MapRegion(f, -1, prot, flags, 0)
(rr) 
0x000000000048cd22	49		return MapRegion(f, -1, prot, flags, 0)

My guess is /proc is locked down on your system in some way.

@rocallahan
Copy link
Collaborator

Do you have /proc mounted with hidepid=1 maybe?

@firelizzard18
Copy link
Author

❯ mount | grep proc
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17427)

I can run it normally and debug with dlv with no problem. It's only when I debug with RR that things get weird. I did run into some weird issues a while back after a system update. I was using a package that called mlock. It worked perfectly fine for more than a year and then started throwing a fit after the update (go-delve/delve#3490). I think the update must have made some kind of security change. I'm wondering if something similar is happening here. Maybe Gentoo or the Kernel now defaults to some more secure profile that causes problems. I'm not sure how to debug that though, since I hardly even know what could have changed.

firelizzard18 added a commit to AccumulateNetwork/accumulate that referenced this issue Nov 5, 2023
* Closes #3464. Sets the supply to 1M ACME by default to prevent negative supply issues.
* Refactors the snapshot indexer to allow mmap to be disabled due to [a bug in rr](rr-debugger/rr#3635).
* Panics if batch commit fails, since that is a fatal error and if returned it can be masked.

Changelog: fix
@firelizzard18
Copy link
Author

I've worked around the mmap issue by using build flags (basically ifdef) to switch out with an implementation that does not use mmap. But now I'm seeing another seemingly similar error on an unrelated project.

rr backtrace
[FATAL src/AutoRemoteSyscalls.cc:734:check_syscall_result()] 
 (task 13696 (rec:13567) at time 2242)
 -> Assertion `false' failed to hold. Syscall openat failed with errno EACCES opening /proc/13588/fd/31
Tail of trace dump:
{
  real_time:865.089360 global_time:2222, event:`SYSCALL: futex' (state:EXITING_SYSCALL) tid:13570, ticks:2119
rax:0x0 rbx:0x0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x80 rdi:0xc00005e948 rbp:0xc00007bd40 rsp:0xc00007bcf8 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00007bd70 r13:0x1 r14:0xc000006e00 r15:0x1 rip:0x478123 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xca fs_base:0xc00005e898 gs_base:0x0
}
{
  real_time:865.089373 global_time:2223, event:`SYSCALL: clock_gettime' (state:ENTERING_SYSCALL) tid:13570, ticks:2229
rax:0xffffffffffffffda rbx:0xc00005e808 rcx:0xffffffffffffffff rdx:0xc00007bd90 rsi:0xc00007bd60 rdi:0x1 rbp:0xc00007bd80 rsp:0xc00007bd58 r8:0x1 r9:0xc0000120ac r10:0x1 r11:0x246 r12:0xc00007bd70 r13:0x1 r14:0xc000006e00 r15:0x1 rip:0x6ffff030 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe4 fs_base:0xc00005e898 gs_base:0x0
}
{
  real_time:865.089383 global_time:2224, event:`SYSCALL: clock_gettime' (state:EXITING_SYSCALL) tid:13570, ticks:2229
rax:0x0 rbx:0xc00005e808 rcx:0xffffffffffffffff rdx:0xc00007bd90 rsi:0xc00007bd60 rdi:0x1 rbp:0xc00007bd80 rsp:0xc00007bd58 r8:0x1 r9:0xc0000120ac r10:0x1 r11:0x246 r12:0xc00007bd70 r13:0x1 r14:0xc000006e00 r15:0x1 rip:0x6ffff030 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe4 fs_base:0xc00005e898 gs_base:0x0
  { tid:13570, addr:0xc00007bd60, length:0x10 }
}
{
  real_time:865.089396 global_time:2225, event:`SYSCALL: futex' (state:ENTERING_SYSCALL) tid:13570, ticks:2323
rax:0xffffffffffffffda rbx:0x0 rcx:0xffffffffffffffff rdx:0x0 rsi:0x80 rdi:0xc00005e948 rbp:0xc00007bd40 rsp:0xc00007bcf8 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00007bd70 r13:0x1 r14:0xc000006e00 r15:0x1 rip:0x478123 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xca fs_base:0xc00005e898 gs_base:0x0
}
{
  real_time:865.089413 global_time:2226, event:`SYSCALL: openat' (state:EXITING_SYSCALL) tid:13567, ticks:729267
rax:0x4 rbx:0xffffffffffffff9c rcx:0xffffffffffffffff rdx:0x80000 rsi:0xc000036540 rdi:0xffffffffffffff9c rbp:0xc00012b510 rsp:0xc00012b4d0 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc000036540 r13:0x0 r14:0xc000007180 r15:0x7ffffffffffff rip:0x40714e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x101 fs_base:0xf2d030 gs_base:0x0
}
{
  real_time:865.089433 global_time:2227, event:`SYSCALL: clock_gettime' (state:ENTERING_SYSCALL) tid:13567, ticks:729290
rax:0xffffffffffffffda rbx:0xf2cfa0 rcx:0xffffffffffffffff rdx:0x7ffe258e7ab8 rsi:0x7ffe258e7a80 rdi:0x1 rbp:0x7ffe258e7aa8 rsp:0x7ffe258e7a78 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x7ffe258e7a98 r13:0x0 r14:0xf2bae0 r15:0x7ffffffffffff rip:0x6ffff030 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe4 fs_base:0xf2d030 gs_base:0x0
}
{
  real_time:865.089444 global_time:2228, event:`SYSCALL: clock_gettime' (state:EXITING_SYSCALL) tid:13567, ticks:729290
rax:0x0 rbx:0xf2cfa0 rcx:0xffffffffffffffff rdx:0x7ffe258e7ab8 rsi:0x7ffe258e7a80 rdi:0x1 rbp:0x7ffe258e7aa8 rsp:0x7ffe258e7a78 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0x7ffe258e7a98 r13:0x0 r14:0xf2bae0 r15:0x7ffffffffffff rip:0x6ffff030 eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe4 fs_base:0xf2d030 gs_base:0x0
  { tid:13567, addr:0x7ffe258e7a80, length:0x10 }
}
{
  real_time:865.089459 global_time:2229, event:`SYSCALL: fcntl' (state:ENTERING_SYSCALL) tid:13567, ticks:729453
rax:0xffffffffffffffda rbx:0x4 rcx:0xffffffffffffffff rdx:0x0 rsi:0x3 rdi:0x4 rbp:0xc00012b510 rsp:0xc00012b4d0 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006a720 r13:0x0 r14:0xc000007180 r15:0xfffffffffff rip:0x40714e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0xf2d030 gs_base:0x0
}
{
  real_time:865.089468 global_time:2230, event:`SYSCALL: fcntl' (state:EXITING_SYSCALL) tid:13567, ticks:729453
rax:0x8000 rbx:0x4 rcx:0xffffffffffffffff rdx:0x0 rsi:0x3 rdi:0x4 rbp:0xc00012b510 rsp:0xc00012b4d0 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006a720 r13:0x0 r14:0xc000007180 r15:0xfffffffffff rip:0x40714e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0xf2d030 gs_base:0x0
}
{
  real_time:865.089481 global_time:2231, event:`SYSCALL: fcntl' (state:ENTERING_SYSCALL) tid:13567, ticks:729500
rax:0xffffffffffffffda rbx:0x4 rcx:0xffffffffffffffff rdx:0x8800 rsi:0x4 rdi:0x4 rbp:0xc00012b510 rsp:0xc00012b4d0 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006a720 r13:0x0 r14:0xc000007180 r15:0xfffffffffff rip:0x40714e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0xf2d030 gs_base:0x0
}
{
  real_time:865.089489 global_time:2232, event:`SYSCALL: fcntl' (state:EXITING_SYSCALL) tid:13567, ticks:729500
rax:0x0 rbx:0x4 rcx:0xffffffffffffffff rdx:0x8800 rsi:0x4 rdi:0x4 rbp:0xc00012b510 rsp:0xc00012b4d0 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006a720 r13:0x0 r14:0xc000007180 r15:0xfffffffffff rip:0x40714e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0xf2d030 gs_base:0x0
}
{
  real_time:865.089503 global_time:2233, event:`SYSCALL: epoll_ctl' (state:ENTERING_SYSCALL) tid:13567, ticks:729569
rax:0xffffffffffffffda rbx:0x5 rcx:0xffffffffffffffff rdx:0x4 rsi:0x1 rdi:0x5 rbp:0xc00012b550 rsp:0xc00012b4f8 r8:0x0 r9:0x0 r10:0xc00012b53c r11:0x246 r12:0xc00006a720 r13:0x0 r14:0xc000007180 r15:0xfffffffffff rip:0x40714e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe9 fs_base:0xf2d030 gs_base:0x0
}
{
  real_time:865.089511 global_time:2234, event:`SYSCALL: epoll_ctl' (state:EXITING_SYSCALL) tid:13567, ticks:729569
rax:0xffffffffffffffff rbx:0x5 rcx:0xffffffffffffffff rdx:0x4 rsi:0x1 rdi:0x5 rbp:0xc00012b550 rsp:0xc00012b4f8 r8:0x0 r9:0x0 r10:0xc00012b53c r11:0x246 r12:0xc00006a720 r13:0x0 r14:0xc000007180 r15:0xfffffffffff rip:0x40714e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0xe9 fs_base:0xf2d030 gs_base:0x0
}
{
  real_time:865.089524 global_time:2235, event:`SYSCALL: fcntl' (state:ENTERING_SYSCALL) tid:13567, ticks:729627
rax:0xffffffffffffffda rbx:0x4 rcx:0xffffffffffffffff rdx:0x0 rsi:0x3 rdi:0x4 rbp:0xc00012b510 rsp:0xc00012b4d0 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006a720 r13:0x0 r14:0xc000007180 r15:0xfffffffffff rip:0x40714e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0xf2d030 gs_base:0x0
}
{
  real_time:865.089532 global_time:2236, event:`SYSCALL: fcntl' (state:EXITING_SYSCALL) tid:13567, ticks:729627
rax:0x8800 rbx:0x4 rcx:0xffffffffffffffff rdx:0x0 rsi:0x3 rdi:0x4 rbp:0xc00012b510 rsp:0xc00012b4d0 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006a720 r13:0x0 r14:0xc000007180 r15:0xfffffffffff rip:0x40714e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0xf2d030 gs_base:0x0
}
{
  real_time:865.089545 global_time:2237, event:`SYSCALL: fcntl' (state:ENTERING_SYSCALL) tid:13567, ticks:729674
rax:0xffffffffffffffda rbx:0x4 rcx:0xffffffffffffffff rdx:0x8000 rsi:0x4 rdi:0x4 rbp:0xc00012b510 rsp:0xc00012b4d0 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006a720 r13:0x0 r14:0xc000007180 r15:0xfffffffffff rip:0x40714e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0xf2d030 gs_base:0x0
}
{
  real_time:865.089553 global_time:2238, event:`SYSCALL: fcntl' (state:EXITING_SYSCALL) tid:13567, ticks:729674
rax:0x0 rbx:0x4 rcx:0xffffffffffffffff rdx:0x8000 rsi:0x4 rdi:0x4 rbp:0xc00012b510 rsp:0xc00012b4d0 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc00006a720 r13:0x0 r14:0xc000007180 r15:0xfffffffffff rip:0x40714e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x48 fs_base:0xf2d030 gs_base:0x0
}
{
  real_time:865.089568 global_time:2239, event:`SYSCALL: fstat' (state:ENTERING_SYSCALL) tid:13567, ticks:729884
rax:0xffffffffffffffda rbx:0x4 rcx:0xffffffffffffffff rdx:0x0 rsi:0xc000140448 rdi:0x4 rbp:0xc00012b4d0 rsp:0xc00012b490 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc000140410 r13:0x0 r14:0xc000007180 r15:0x3ffffffffffffff rip:0x40714e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x5 fs_base:0xf2d030 gs_base:0x0
}
{
  real_time:865.089579 global_time:2240, event:`SYSCALL: fstat' (state:EXITING_SYSCALL) tid:13567, ticks:729884
rax:0x0 rbx:0x4 rcx:0xffffffffffffffff rdx:0x0 rsi:0xc000140448 rdi:0x4 rbp:0xc00012b4d0 rsp:0xc00012b490 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc000140410 r13:0x0 r14:0xc000007180 r15:0x3ffffffffffffff rip:0x40714e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x5 fs_base:0xf2d030 gs_base:0x0
  { tid:13567, addr:0xc000140448, length:0x90 }
}
{
  real_time:865.089593 global_time:2241, event:`SYSCALL: mmap' (state:ENTERING_SYSCALL) tid:13567, ticks:730157
rax:0xffffffffffffffda rbx:0x0 rcx:0xffffffffffffffff rdx:0x1 rsi:0x1000 rdi:0x0 rbp:0xc00012b4d0 rsp:0xc00012b490 r8:0x4 r9:0x0 r10:0x1 r11:0x246 r12:0x0 r13:0x0 r14:0xc000007180 r15:0x3ffffffffffffff rip:0x40714e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0xf2d030 gs_base:0x0
}
{
  real_time:865.089791 global_time:2242, event:`SYSCALL: mmap' (state:EXITING_SYSCALL) tid:13567, ticks:730157
rax:0x7fd7b442d000 rbx:0x0 rcx:0xffffffffffffffff rdx:0x1 rsi:0x1000 rdi:0x0 rbp:0xc00012b4d0 rsp:0xc00012b490 r8:0x4 r9:0x0 r10:0x1 r11:0x246 r12:0x0 r13:0x0 r14:0xc000007180 r15:0x3ffffffffffffff rip:0x40714e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x9 fs_base:0xf2d030 gs_base:0x0
  { map_file:"/home/firelizzard/.cache/go-build/07/07487a3186c01f97d75afcf23b2c83da4d442754d51e5055304fe937f2c8684e-d", addr:0x7fd7b442d000, length:0x1000, prot_flags:"r--s", file_offset:0x0, device:2051, inode:37930354, data_file:"", data_offset:0x0, file_size:0xb2b }
  { tid:13567, addr:0x7fd7b442d000, length:0xb2b }
}
{
  real_time:865.089851 global_time:2243, event:`SYSCALL: openat' (state:ENTERING_SYSCALL) tid:13567, ticks:755123
rax:0xffffffffffffffda rbx:0xffffffffffffff9c rcx:0xffffffffffffffff rdx:0x80000 rsi:0xc0000328d0 rdi:0xffffffffffffff9c rbp:0xc00012b058 rsp:0xc00012b018 r8:0x0 r9:0x0 r10:0x0 r11:0x246 r12:0xc0000328d0 r13:0x0 r14:0xc000007180 r15:0x1ffffffff rip:0x40714e eflags:0x246 cs:0x33 ss:0x2b ds:0x0 es:0x0 fs:0x0 gs:0x0 orig_rax:0x101 fs_base:0xf2d030 gs_base:0x0
}
[FATAL src/log.cc:444:emergency_debug()] (session doesn't look interactive, aborting emergency debugging)
=== Start rr backtrace:
rr(_ZN2rr13dump_rr_stackEv+0x41)[0x55ae21fd1b61]
rr(_ZN2rr15notifying_abortEv+0xe)[0x55ae21fd309e]
rr(+0x1bdf0f)[0x55ae21fe8f0f]
rr(+0xabe43)[0x55ae21ed6e43]
rr(+0xac8cb)[0x55ae21ed78cb]
rr(_ZN2rr18AutoRemoteSyscalls20check_syscall_resultElib+0x186)[0x55ae21e86896]
rr(_ZN2rr18AutoRemoteSyscalls18finish_direct_mmapENS_10remote_ptrIvEEmiiRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEilR4statRS8_+0x1fd)[0x55ae21e878bd]
rr(+0x139e71)[0x55ae21f64e71]
rr(+0x13b377)[0x55ae21f66377]
rr(+0x13fe37)[0x55ae21f6ae37]
rr(_ZN2rr13ReplaySession28setup_replay_one_trace_frameEPNS_10ReplayTaskE+0x7b7)[0x55ae21f62367]
rr(_ZN2rr13ReplaySession11replay_stepERKNS0_15StepConstraintsE+0x9c)[0x55ae21f6254c]
rr(_ZN2rr14ReplayTimeline19replay_step_forwardENS_10RunCommandE+0xc4)[0x55ae21f7bd94]
rr(_ZN2rr9GdbServer14debug_one_stepERNS_10GdbRequestE+0x3c5)[0x55ae21ec3e25]
rr(_ZN2rr9GdbServer12serve_replayERKNS0_15ConnectionFlagsE+0x69b)[0x55ae21ec526b]
rr(+0x12c61a)[0x55ae21f5761a]
rr(_ZN2rr13ReplayCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0x63f)[0x55ae21f58e6f]
rr(main+0x1ad)[0x55ae21e7183d]
/usr/lib64/libc.so.6(+0x23dda)[0x7f0d0b7ffdda]
/usr/lib64/libc.so.6(__libc_start_main+0x85)[0x7f0d0b7ffe95]
rr(_start+0x21)[0x55ae21e71a11]
=== End rr backtrace
2024-05-09T11:48:25-05:00 error layer=dap runtime error: EOF
Unable to retrieve goroutines: write tcp 127.0.0.1:46490->127.0.0.1:13588: write: broken pipe
Detaching and terminating target process
2024-05-09T11:48:44-05:00 error layer=dap detach returned error: could not detach process 0: write tcp 127.0.0.1:46490->127.0.0.1:13588: write: broken pipe

@khuey
Copy link
Collaborator

khuey commented May 9, 2024

Does this unrelated project use ptrace?

@firelizzard18
Copy link
Author

firelizzard18 commented May 9, 2024

Nope, it's just a basic Go project (here). I'm definitely not doing anything with ptrace, and I'm fairly certain neither of the libraries I'm calling do. I suppose it's possible the Go runtime is. I can debug fine, but RR fails when it tries to replay a nil pointer exception. But when I make a trivial program with a nil pointer dereference (func main() { var x *int = nil; *x = 1}), it can handle that fine.

I found this post that talks about profs having non-standard permissions, and references PTRACE_MODE_READ_FSCREDS. I've been poking around at processes. For processes like my shell, ls -l /proc/${PID}/fd works:

❯ ls -lh /proc/17190/fd                                      
total 0
lrwx------ 1 firelizzard firelizzard 64 May  9 11:55 0 -> /dev/pts/2
lrwx------ 1 firelizzard firelizzard 64 May  9 12:09 1 -> /dev/pts/2
lrwx------ 1 firelizzard firelizzard 64 May  9 12:09 2 -> /dev/pts/2
lrwx------ 1 firelizzard firelizzard 64 May  9 12:09 10 -> /dev/pts/2

But when I look for RR processes:

fireliz+   22959  0.0  0.0  20788 13192 ?        S    12:08   0:00 rr replay --dbgport=0 /home/firelizzard/.local/share/rr/__debug_bin4120815066-0
fireliz+   22960  0.0  0.0 1465820 8640 ?        ts   12:08   0:00 [rr:__debug_bin4]
fireliz+   22961  0.0  0.0   8816   580 ?        t    12:08   0:00 [rr:__debug_bin4]
fireliz+   22962  0.0  0.0   8816   580 ?        t    12:08   0:00 [rr:__debug_bin4]
fireliz+   22974  0.0  0.0 1465820 8640 ?        t    12:08   0:00 [rr:__debug_bin4]
fireliz+   22975  0.0  0.0 1465820 8640 ?        t    12:08   0:00 [rr:__debug_bin4]
fireliz+   22976  0.0  0.0 1465820 8640 ?        t    12:08   0:00 [rr:__debug_bin4]

And run ls -l /proc/${PID}/fd on one of those, I can't read any of the 'symlinks':

ls: cannot read symbolic link '/proc/22976/fd/0': Permission denied
ls: cannot read symbolic link '/proc/22976/fd/1': Permission denied
ls: cannot read symbolic link '/proc/22976/fd/2': Permission denied
ls: cannot read symbolic link '/proc/22976/fd/3': Permission denied
...

I run into the same issue with procfs with the trivial program (permission denied on the procfs 'symlinks') but it replays fine, so either the permission denied is not the real issue, or there's something different about how RR is handling these two cases.

@firelizzard18
Copy link
Author

I get the same error running dlv debug --backend rr ./cmd/generate-self on this branch (if I execute (dlv) continue).

@khuey
Copy link
Collaborator

khuey commented May 9, 2024

If you set /proc/sys/kernel/yama/ptrace_scope to 0 does your workload run under rr?

@firelizzard18
Copy link
Author

/proc/sys/kernel/yama doesn't exist on my system. My kernel config is custom so I can enable config options if that would help.

@khuey
Copy link
Collaborator

khuey commented May 9, 2024

Is CONFIG_SECURITY_YAMA not set for your config?

@firelizzard18
Copy link
Author

Correct, it was not set. For the most part I'm using the default config, plus the features I need for my hardware and whatever other features I specifically wanted. I guess Yama defaults to disabled? I set it, rebuilt, and rebooted. ptrace_scope defaulted to 1 so I set it to 0 but that didn't help. I'll probably unset CONFIG_SECURITY_YAMA to avoid changing multiple variables at once.

@firelizzard18
Copy link
Author

@aarzilli Have you by chance run into an issue like this with delve and RR? I don't expect you to provide support for a crash in RR, but I thought you might have seen this before.

@aarzilli
Copy link
Contributor

I haven't. If you want to test with the latest version of rr you need to apply a patch to delve that hasn't been merged yet: go-delve/delve#3718

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants