Mutation testing of PUC Rio Lua stucks on 1406'th mutant out of 1841

[ligurio]

How to reproduce

• download lua source code (https://www.lua.org/ftp/lua-5.4.3.tar.gz) and unpack it
• update CC options in src/Makefile:

--- src/Makefile        2021-04-24 18:38:42.314883151 +0300
+++ src/Makefile.orig   2021-04-24 18:38:35.026873188 +0300
@@ -6,7 +6,7 @@
 # Your platform. See PLATS for possible values.
 PLAT= guess
 
-CC= clang -fembed-bitcode -g
+CC= gcc -std=gnu99
 CFLAGS= -O2 -Wall -Wextra -DLUA_COMPAT_5_3 $(SYSCFLAGS) $(MYCFLAGS)
 LDFLAGS= $(SYSLDFLAGS) $(MYLDFLAGS)
 LIBS= -lm $(SYSLIBS) $(MYLIBS)

• change dir to src and build Lua interpreter with make
• run Mull in src directory: mull-cxx -keep-executable -mutate-only -output=lua.mutated -compilation-flags="-isystem /usr/include/linux/" -linker-flags="-lm -Wl,-E -ldl" lua
• download lua-Harness (git clone https://framagit.org/fperrad/lua-Harness)
• set LUA env variable with path to mutated Lua binary (export LUA=/home/sergeyb/sources/lua-5.4.3/src/lua.mutated)
• run testing: cd lua-Harness/test_lua && mull-runner $LUA -test-program=prove -- --exec="$LUA" 000-sanity.t

Expected

successful testing

Actual

testing stuck on 1406'th mutant (-debug option doesn't provide more details):

$ mull-runner $LUA --debug -test-program=prove -- --exec="$LUA" 000-sanity.t 
[debug] Diagnostics: Debug Mode enabled. Debug-level messages will be printed.
[info] Warm up run (threads: 1)
       [################################] 1/1. Finished in 62ms
[info] Baseline run (threads: 1)
       [################################] 1/1. Finished in 55ms
[info] Running mutants (threads: 8)
       [########################--------] 1406/1841
<process is stalled>

Everything is ok when echo used as a test program:
$ mull-runner $LUA --debug -test-program=echo

Versions

sergeyb@pony:~/sources/lua-5.4.3$ mull-cxx --version
Mull: LLVM-based mutation testing
https://github.com/mull-project/mull
Version: 0.10.0-pr855
Commit: 84b5d8b
Date: 23 Apr 2021
LLVM: 11.0.0

[ligurio]

it's interesting that mutated binary segfaults:

$ ./lua.mutated 
Lua 5.4.3  Copyright (C) 1994-2021 Lua.org, PUC-Rio
> a = 1                        
Segmentation fault  

[AlexDenisov]

The smallest crasher I could find so far:

mull-cxx \
  -keep-executable \
  -mutate-only \
  -output=lua.mutated \
  -mutators=cxx_add_to_sub \
  -linker-flags="-lm -Wl,-E -ldl" \
  -include-path=lvm.c \
  lua 

it generates two mutations that lead lua.mutated to the crash,

[AlexDenisov]

@ligurio I've finally managed to find the root cause behind this crash. I also have a rough idea how to fix it, so let's hope we can close this issue soon 🤞

[AlexDenisov]

(I should've linked the root cause explanation #961)

[AlexDenisov]

Turned out, I don't yet know how to implement a proper fix, so for the time being I'll add a check for functions using jump tables to not miscompile them.
I double checked, and with #963 merged you can build and run Mull against Lua.

I'll release a hotfix version in the next couple of days and will close this issue.
The root cause will be tracker separately in #961.

[ligurio]

I'm trying to reproduce an issue with 0.15.1 and stuck with a problem "error: Unknown attribute kind (68) (Producer: 'LLVM12.0.1' Reader: 'LLVM 10.0.0')":

Change src/Makefile:

--- Makefile.orig       2022-01-31 13:55:04.759326427 +0300
+++ Makefile    2022-01-31 13:55:16.667338525 +0300
@@ -6,8 +6,9 @@
 # Your platform. See PLATS for possible values.
 PLAT= guess
 
-CC= gcc -std=gnu99
-CFLAGS= -O2 -Wall -Wextra -DLUA_COMPAT_5_3 $(SYSCFLAGS) $(MYCFLAGS)
+CC= clang-12 -fembed-bitcode -g
+MULL_FLAGS= -fexperimental-new-pass-manager -fpass-plugin=/usr/lib/mull-ir-frontend-12 -grecord-command-line
+CFLAGS= -O2 -Wall -Wextra -DLUA_COMPAT_5_3 $(SYSCFLAGS) $(MYCFLAGS) $(MULL_FLAGS)
 LDFLAGS= $(SYSLDFLAGS) $(MYLDFLAGS)
 LIBS= -lm $(SYSLIBS) $(MYLIBS)
 

Compile Lua:

sergeyb@pony:~/sources/lua-5.4.3/src$ make
Guessing Linux
make[1]: Entering directory '/home/sergeyb/sources/lua-5.4.3/src'
make all SYSCFLAGS="-DLUA_USE_LINUX" SYSLIBS="-Wl,-E -ldl"
make[2]: Entering directory '/home/sergeyb/sources/lua-5.4.3/src'
make[2]: Nothing to be done for 'all'.
make[2]: Leaving directory '/home/sergeyb/sources/lua-5.4.3/src'
make[1]: Leaving directory '/home/sergeyb/sources/lua-5.4.3/src'
sergeyb@pony:~/sources/lua-5.4.3/src$ mull-cxx-12 -keep-executable -mutate-only -output=lua.mutated -compilation-flags="-isystem /usr/lib/llvm-12/lib/clang/12.0.1/include" -linker-flags="-lm -Wl,-E -ldl" -linker=clang++ lua
...
[info] Compiling original code (threads: 8)                                                                                             
       [################################] 33/33. Finished in 798ms
[info] Link mutated program (threads: 1)                                                                                                
[error] Cannot link program
status: Failed                                                                                                                          
time: 625ms       
exit: 1                                                                                                                                 
command: clang++ -lm -Wl,-E -ldl /tmp/mull-4eb88e.bc /tmp/mull-505f3b.bc /tmp/mull-1b1663.bc /tmp/mull-4710a4.bc /tmp/mull-76d4ed.bc /tm
p/mull-52e92a.bc /tmp/mull-4c5b26.bc /tmp/mull-ccd802.bc /tmp/mull-bbcec5.bc /tmp/mull-ce3b90.bc /tmp/mull-8da362.bc /tmp/mull-ba3096.bc
 /tmp/mull-34d564.bc /tmp/mull-54129b.bc /tmp/mull-5ab443.bc /tmp/mull-bbb231.bc /tmp/mull-1d9e2f.bc /tmp/mull-b17f5b.bc /tmp/mull-14063
3.bc /tmp/mull-85ea77.bc /tmp/mull-1b227b.bc /tmp/mull-444a3d.bc /tmp/mull-886719.bc /tmp/mull-5a750a.bc /tmp/mull-257860.bc /tmp/mull-3
75e2e.bc /tmp/mull-63fd34.bc /tmp/mull-1c6199.bc /tmp/mull-521c92.bc /tmp/mull-1ed4a6.bc /tmp/mull-33a4a1.bc /tmp/mull-698274.bc /tmp/mu
ll-8cc9c6.bc -o lua.mutated
stdout: 
stderr: error: Unknown attribute kind (68) (Producer: 'LLVM12.0.1' Reader: 'LLVM 10.0.0')
1 error generated.
error: Unknown attribute kind (68) (Producer: 'LLVM12.0.1' Reader: 'LLVM 10.0.0')
...

@AlexDenisov, could you tell me how to resolve it?

[AlexDenisov]

@ligurio with the IR frontend, you no longer need mull-cxx. Just run mull-runner-12 against lua executable - it has all the mutant in it already.

[ligurio]

@AlexDenisov Nice! Finally it works, thanks! :)

[info] Mutation score: 13%
[info] Total execution time: 130378ms