A 'testFileMapping' to enforce and speedup mutation testing for test policies with fixed test file naming

[sebiniemann]

Is your feature request related to a problem? Please describe.
As outlined in the discussions at #3595 and #4142, we are also organizing our unit tests in a manner where each specific test file corresponds to a particular system under test. Likewise, there is also a fixed naming scheme to keep everything well-organized for the entire engineering team. However, we do not use jest, but a tap based framework where there is no such option like --findRelatedTests and dependencyExtractor.

For us, the desired coverage of a system under test should be exclusively derived from its dedicated test file, ensuring comprehensive testing of all aspects of the system under test. Coverage from other test files is considered an unintended side-effect.

Enabling the possibility to configure the intended mapping between test files and systems under test would assist us in achieving two key objectives:

• Enforcement of such a testing policy, ensuring that the corresponding test file(s) independently cover all aspects of the system under test.
• Optimization of mutation testing speed by executing only the intended test file(s) per mutation.

Describe the solution you'd like
The addition of a testFileMapping configuration feature, that associates mutated files with their corresponding test file(s). The existing mapping which was collected during the dryRun phase could serve as a fallback for files not explicitly listed in the testFileMapping option.

To accommodate various use cases, the file to be tested could be represented by a regular expression, especially supporting regex groups. The corresponding test files notation could utilize these groups, acting as a filter on the fallback mapping derived from the dryRun phase.

An implementation of this logic could resemble the following:

const config = JSON.parse(`{
  "testFileMapping": [{
    "file": "^custom\.js$",
    "testFile": "test/xyz/custom.test.js"
  }, {
    "file": "^(.+?)\.js$",
    "testFile": "test/$1.test.js"
  }]
}`);

// The file under mutation testing
const file = 'abc.js' 

// Collection from the dryRun phase
let testFiles = [
  'test/abc.test.js',
  'test/def.test.js',
];

if ('testFileMapping' in config) {
  for (const candidate of config.testFileMapping) {
    const fileRegex = new RegExp(candidate.file);

    if (fileRegex.test(file)) {
      const testFileRegex = new RegExp(file.replace(fileRegex, candidate.testFile));

      testFiles = testFiles.filter((x) => testFileRegex.test(x));
      break;
    }
  }
}

console.log(testFiles);

// Proceeding with mutation testing of the "file" using "testFiles"

We are open to contributing a merge request if this proposal aligns with your intentions 😃

Describe alternatives you've considered
We tried out the solution proposed in #3595 by using the programmatic api. In terms of functionality, it achieved our goal and also reduced the testing duration to a third. However, Stryker had to be started for each individual system under test, which not only creates quite the overhead, but also a large number of individual reports that would then have to be merged back into an overall report using additonal (yet to be implemented) post-processing steps.

We have also looked at writing our own plugin, but it seems that this would require much more than just the filter, which has so far prevented us from doing so.

[sebiniemann]

In case someone is interested in our current workaround using the programmatic api, here is a minimal setup of our script.

import * as extStrykerMutatorCore from '@stryker-mutator/core';

// Collection of all files under mutation testing
const files = [/* ... */];

for (const file of files) {
  // Mapping the name of the file under mutation to its corresponding test files
  let testFiles = [/* ... */]; 

  const stryker = new extStrykerMutatorCore.Stryker({
    // Makes the sum of the reports of the individual runs look more like a collective report of one big run
    clearTextReporter: {
      logTests: false,
      reportTests: false,
      reportScoreTable: false,
    },
    // The test runner we use
    tap: {
      nodeArgs: ['--test-reporter', 'tap'],
      testFiles,
    },
    testRunner: 'tap',
    // See above, removes info messages between the reports of the individual runs to make it look more like one big run
    logLevel: 'warn',
    mutate: [file],
    reporters : ['clear-text'],
  });

  await stryker.runMutationTest();
}

In case were testFiles will always contain exactly one entry, the tap runner provided no added value for us and it was faster to use the commandRunner and set coverageAnalysis to off.

However, this may not be true for test runners that also consider the individual tests within the test file during the dryRun, which could therefore still achieve a speedup compared to the commandRunner. This is why the example above lists the test runner we normally use, so as not to give the impression that you need to use the commandRunner for this.

---

To benefit from the incremental mode, while creating a separate stryker run for each file, it is necessary to create the incrementalFile in dependency of the file to be tested. To do this, we have added the following to the configuration:

incremental: true,
incrementalFile: `./stryker-incremental-${hash(file)}.json`,

[sebiniemann]

We've continued to think about how one could integrate this feature more easily. Since the script mentioned above already largely helps to implement this feature, one could consider to keep a major part of the function to userland instead of integrating it directly into this library.

For the reported coverage result, it would be useful if we could collect the results of each run and pass them collectively to the reporters. For this to work, the reporters would need to be officially/intentionally callable from userland – we have yet to check if this is already possible.

To address the requests from the other issues noted above, how aboud explaining in the documentation how one can implement this themselves. What do you think?