Test runs are not consistent

[jeznag]

In the same codebase, I'm getting wildly different mutant kill scores. Something is not working right.

[jeznag]

@nicojs @simondel this is probably because the tape object stream listener sticks around within the child process. Does stryker terminate the child process after each mutant is run?

Another question:
Do you know how to catch errors that tape generates?

        // requiring the tape file is enough to execute the tests
          // NB - tape-catch is required otherwise the whole thing blows up
          // here if the test throws an exception
          try {
            require(testFile.path);
          } catch (error) {
            resolve({
              status: RunStatus.Error,
              tests: [],
              errorMessages: [error]
            });
          }

doesn't actually catch the error:

    expect.deepEqual(actualOutput.attributeScores, expectedOutput.attributeScores);
                                 ^

TypeError: Cannot read property 'attributeScores' of undefined
    at Test.<anonymous> (/Users/jeremy/Documents/dev/sideprojects/empath-sentiment-analysis/.stryker-tmp/9858812/sandbox341979/test/tape/analyseSentiment.js:16:22)
    at Test.bound [as _cb] (/Users/jeremy/Documents/dev/sideprojects/empath-sentiment-analysis/node_modules/tape/lib/test.js:66:32)
    at Test.run (/Users/jeremy/Documents/dev/sideprojects/empath-sentiment-analysis/node_modules/tape/lib/test.js:85:10)
    at Test.bound [as run] (/Users/jeremy/Documents/dev/sideprojects/empath-sentiment-analysis/node_modules/tape/lib/test.js:66:32)
    at Immediate.next (/Users/jeremy/Documents/dev/sideprojects/empath-sentiment-analysis/node_modules/tape/lib/results.js:71:15)
    at runCallback (timers.js:637:20)
    at tryOnImmediate (timers.js:610:5)
    at processImmediate [as _immediateCallback] (timers.js:582:5)

That's why I tried object stream originally. Can't figure out how to do it.

[nicojs]

this is probably because the tape object stream listener sticks around within the child process. Does stryker terminate the child process after each mutant is run?

No, Styker does not terminate the child process after each test run. This is for performance reasons: spawning of a new process is very expensive. Stryker does, however, guarantee that the test runner will only be used for one test run at a time.

Does 'tape' keep state about a previous test run? If it does, that would be bad. The preferred way of fixing this would be fixing that issue in tape. You should be able to instantiate a new instance of the Tape test runner with its own isolated state (like you can with mocha) and go to use that.

An other way (hack) would be to clear Tape and all its friends from the cache (for tape itself it would be delete require.cache[require.resolve('tape')]) and than re-require it yourself for each test run (so put require('tape') inside the run() after you've cleared it from cache). This is pretty tricky however, every file should be cleared one by one. For example: if tape itself does something like require('./myObjectStream'), than you should also clear that (for example: delete require.cache[path.join(path.dirname(require.resolve('tape')), 'myObjectStream.js'). As far as i know, there is no way of clearing the entire dependency tree, so it would be hard coded in stryker-tape-runner and will break when'./myObjectStream' gets renamed to './MyObjectStream' for example.

With all of that said. If tape does not keep state, the solution seems simpler.

tape.createStream({ objectMode: true, port: this.port })
  .on('data', ...)
  .on('end', ...)

You can move this out of the run() method and put it in the init (or constructor as it's synchronous code). You would be able to keep the testResults variable as an instance variable of your class and every time run() gets called: clear it (this.testResults = []).

Another question: Do you know how to catch errors that tape generates?

No i don't. Could you create an integration test/ unit test which reproduces and error, than i can take a look. However i do know that try...catch only handles errors that are thrown synchronously. So if tape does something async, the try-catch will not handle that. This is easily tested: is the line after all require(testFile.path);'s code executed before all data is received on the tape object stream?