Enabling GPU on NVIDIA drivers causes crashes

[TheImunityGamer]

Useful information:

Please, try to gather as much of useful information as possible and follow
these instructions:

• Version:

  • Run this command: wayvnc -V
  • wayvnc: v0.6.2-30-g945f590 (master)
  • neatvnc: v0.6.0-31-g56f1c12 (master)
  • aml: v0.3.0-1-gede4249 (master)

• Try to reproduce while capturing a trace log:

  • wayvnc -Ltrace | tee wayvnc-crash.log

TRACE: ../src/main.c: 234: Registering new wlr_output_power_manager
TRACE: ../src/main.c: 224: Registering new xdg_output_manager
TRACE: ../src/main.c: 205: Registering new output 49
TRACE: ../src/output.c: 222: Output 49 name: HDMI-A-1
TRACE: ../src/output.c: 231: Output 49 description: Samsung Electric Company LS24AG30x H4PR700788 (HDMI-A-1)
TRACE: ../src/output.c: 274: Output HDMI-A-1 power state changed to ON
Info: Capturing output HDMI-A-1
Info: >> Samsung Electric Company LS24AG30x H4PR700788 (HDMI-A-1) 1920x1080+0x0 Power:ON
DEBUG: ../subprojects/neatvnc/src/server.c: 1360: Trying address: 0.0.0.0
DEBUG: ../subprojects/neatvnc/src/server.c: 1380: Successfully bound to address
Info: Listening for connections on 0.0.0.0:25565
DEBUG: ../src/ctl-server.c: 751: Initializing wayvncctl socket: /run/user/1000/wayvncctl
Info: New client connection from <redacted>: 0x5559d9eac8c0 (ref 1)
Info: Starting screen capture
DEBUG: ../src/main.c: 1100: Client connected, new client count: 1
DEBUG: ../src/ctl-server.c: 879: Enqueueing client-connected event: {"id":"1","hostname":"<redacted>","username":null,"seat":"seat0","connection_count":1}
DEBUG: ../src/ctl-server.c: 906: Enqueued client-connected event for 0 clients
Info: Choosing zrle encoding for client 0x5559d9eac8c0

• Get the stack trace:
  • If have coredumpctl, you can gather the stack trace after a crash using
    coredumpctl gdb wayvnc and then run bt full to obtain the stack trace.
  • Otherwise, you can either locate the core file and load it into gdb or run
    wayvnc in gdb like so:
    • gdb --args wayvnc -Ltrace
  • If the lines mentioning wayvnc, neatvnc or aml have ??, please compile
    wayvnc and those other projects from source with debug symbols and try
    again.

#0  0x00007f8648a8e83c in  () at /usr/lib/libc.so.6
#1  0x00007f8648a3e668 in raise () at /usr/lib/libc.so.6
#2  0x00007f8648a264b8 in abort () at /usr/lib/libc.so.6
#3  0x00007f8648a263dc in  () at /usr/lib/libc.so.6
#4  0x00007f8648a36d26 in  () at /usr/lib/libc.so.6
#5  0x00007f8649016dc3 in zrle_encoder_do_work (obj=0x5559d9ec23e0) at ../subprojects/neatvnc/src/zrle.c:354
        self = 0x5559d9eec440
        rc = -1
        fb = 0x5559d9eb9460
        __PRETTY_FUNCTION__ = "zrle_encoder_do_work"
        buffer_size = 0
        dst = {data = 0x0, len = 0, cap = 0}
        src_fmt = {bits_per_pixel = 32 ' ', depth = 24 '\030', big_endian_flag = 0 '\000', true_colour_flag = 1 '\001', red_max = 255, green_max = 255, blue_max = 255, red_shift = 16 '\020', green_shift = 8 '\b', blue_shift = 0 '\000', padding = "\177\000"}
#6  0x00007f8649036904 in worker_fn (context=0x0) at ../subprojects/aml/src/thread-pool.c:111
        work = 0x5559d9ebe3a0
        cb = 0x7f8649016c2c <zrle_encoder_do_work>
        aml = 0x0
        ss = {__val = {18446744067267035135, 0 <repeats 15 times>}}
        __PRETTY_FUNCTION__ = "worker_fn"
#7  0x00007f8648a8c9eb in  () at /usr/lib/libc.so.6
#8  0x00007f8648b10dfc in  () at /usr/lib/libc.so.6

• Describe how to reproduce the problem
  I applied the patch recommended in wayvnc doesn't work with nvidia drivers, because neatvnc doesn't support 24bit pixel input #139 to wlroots in my Hyprland DE. I then had the issue from Segmentation Fault on Arch/Wayland/Hyprland #248 and applied the patch from there to neatvnc. I was able to connect using RealVNC to Hyprland running on a PC with an NVIDIA card. It was in CPU mode so I added --gpu to it, and this error happened.

[any1]

Coercing the software rendering pixel format to a 32 bit one instead of 24 will not change the GPU buffer format into a 32 bit one as well. It's still 24 bits, and those are not currently implemented. Besides, if you're using ZRLE encoding, you won't notice any performance improvement from setting the --gpu flag.

[TheImunityGamer]

Oh well, looks like to make this work, I will have to make a lot of changes. I do not guarantee I will be able to even do it, and I don't guarantee it will be good, but if it works, I could try to make some commits.

[maztheman]

On my machine, nvidia proprietary driver, gbm_bo_map() returns ENOSYS which means its not implemented.