Incremental analysis doesn't work when sharing incremental analysis files between Windows & Mac OS

[mikesmithson]

I've been using incremental analysis with a previous team that was using Mac's exclusively without any issue. I am working on a new team, all remote and we have a mix of engineers using both Mac and Windows. The incremental analysis file generated seems to generate fine when I run pitest on my Mac, push the incremental analysis file into our git repo and then let CI take it from there and our builds are pretty fast. When an engineer I am working with runs pitest locally on his Windows machine, the incremental analysis file has changed, he commits and pushes it and when CI runs another pipeline build, the pitest task doesn't seem to be reading the incremental analysis file and a completely new set of mutation tests are run, which is unacceptably slow. We're trying to figure why this is happening, looked through all of the CLI options we can configure and found nothing that solves this issue. We can only conclude that there is some discrepancy with the Windows operating system that is causing this issue. Here is a sample pitest configuration we are using.

 pitest {
    targetClasses = ["$packagePrefix.*"]
    pitestVersion = "$pitestDistroVersion"
    junit5PluginVersion = "$pitestJunit5PluginVersion"
    threads = (Runtime.getRuntime().availableProcessors() * 0.5).intValue()
    outputFormats = ['XML', 'HTML']
    avoidCallsTo = ['org.slf4j']
    mutators = ['STRONGER']
    timestampedReports = false
    failWhenNoMutations = false
    enableDefaultIncrementalAnalysis = true
    historyInputLocation = pitHistoryFile
    historyOutputLocation = pitHistoryFile
    useClasspathFile = true
}

[szpak]

@mikesmithson Thanks for the report.

@hcoles Have you encountered any similar situation in the non-Gradle projects (creating an incremental analysis file on one OS and using it on another)? I also wonder what is a recommended workflow for the parties to update and commit that file (in general) when working with CI?

[mikesmithson]

@szpak - is there a a way to control the input/output encoding? We've tried configuring the 'outputCharset' and 'inputCharset' properties to be UTF-8 but we get an error and our pitest task fails. I can't recall what it was. I can give you more details tomorrow.

[hcoles]

@szpak I've not had a report of this before, and I've not enocuntered it myself (but I have never attempted to share history files).

In terms of recommended workflow, I describe how I do things here

https://blog.pitest.org/dont-let-your-code-dry/

The history file is hard coded to read/write in UTF-8, so I don't think charsets are the issue. The format of the file is terrible however (quickest thing that could work) and needs to be redone from scratch so it is plain text.

In particular, it is very brittle and sensitive to class changes. Is there a chance that pitestDistroVersion is not the same across machines?

[mikesmithson]

@hcoles - the pitestDistroVersion is a property that is in our gradle file and is the same across machines since it's under source control. For example:

buildscript {
  ext {
       pitestDistroVersion = '1.9.5'
       pitestJunit5PluginVersion = '1.0.0'
   }
}
....
....
 dependencies {
  	testImplementation "org.pitest:pitest:${pitestDistroVersion}"
	testImplementation "org.pitest:pitest-junit5-plugin:${pitestJunit5PluginVersion}"
}
....

We do this because we want renovate to be able to notify us when there is a new pitest distribution and junit 5 plugin published so we are keeping up-to-date.

[mikesmithson]

Anything more you need from me? Is there any traction on this issue? It's really hurting our team's productivity.

[hcoles]

@mikesmithson I will take a look at updating the format of the history store when I get chance. If it is no longer opaque this should help debug the issue.

If your team has an arcmutate licence, send an e-mail to support and this will be added to the priority queue.

[mikesmithson]

as luck would have it, when a team member informed me that he updated the JDK for the project in question from JDK 11 to JDK 17, suddenly everything seemed to work as expected between and Windows OS's. Not sure why, but that may or may not provide some insight.