Prolonged Execution and Job Failures in Mutant Runs with Shards

[ASuciuX]

When executing cargo mutants with 8 shards on our CI (with a diff file that originally took 3:18h without shards), we observed that the longest process took 2:47h for the job with ID 2, while the average execution time for other jobs is 1:15h. While this is an improvement compared to running mutants without shards, it still seems suboptimal (even though it takes a while to run the unmutated build and tests), considering the workload is divided into 8 parts and the expected performance is not achieved. (not even 2x better time results using 8x power)

Additionally, we encountered an issue when using shards where some jobs failed the unmutated tests, while others ran successfully. We attempted to address this by adding the -- -- --test-threads 1 argument to the cargo test command, but, despite this adjustment, the unmutated tests still failed on the job with ID 0.

Reference CI Run with Shards: https://github.com/ASuciuX/stacks-core/actions/runs/7302059398/job/19903435918?pr=35
Reference CI Run without Shards: https://github.com/ASuciuX/stacks-core/actions/runs/7269581208/job/19807417329?pr=27
Actions Code: https://github.com/ASuciuX/actions/blob/test/shards-mutants/mutation-testing/filter-pr/action.yml
Workflow Code: https://github.com/ASuciuX/stacks-core/blob/mutants-next/test-stackslib-with-shards-1/base/.github/workflows/filter-pr-mutants.yml

[sourcefrog]

When executing cargo mutants with 8 shards on our CI (with a diff file that originally took 3:18h without shards), we observed that the longest process took 2:47h for the job with ID 2, while the average execution time for other jobs is 1:15h.

Taking shard 3 as an example:

Found 3 mutants to test
ok       Unmutated baseline in 581.6s build + 1465.4s test
Auto-set test timeout to 2h 2m 6s
caught   testnet/stacks-node/src/mockamoto.rs:994:9: replace MockamotoNode::mine_and_stage_block -> Result<u64, ChainstateError> with Ok(1) in 372.0s build + 524.0s test
unviable stackslib/src/chainstate/nakamoto/miner.rs:431:9: replace NakamotoBlockBuilder::tenure_begin -> Result<ClarityTx<'b, 'b>, Error> with Ok(Default::default()) in 77.2s build
caught   stackslib/src/chainstate/nakamoto/mod.rs:1762:9: replace NakamotoChainState::load_aggregate_public_key -> Result<Point, ChainstateError> with Ok(Default::default()) in 609.8s build + 1251.9s test
3 mutants tested in 1h 5m 9s: 2 caught, 1 unviable

So in this overall run there are only about 3 mutants for each shard to test, but incremental builds are taking up to 10 minutes and the test suite takes up to 20 minutes (or maybe more in shards I didn't inspect.)

With only 3-4 mutants per shard there is necessarily going to be a lot of variability in run time, as some shards will happen to get mutants that are very fast and some shards will be unlucky to get some that are very slow. Shards will give a more even effect when there are many mutants.

The default GitHub workers are not very powerful, so one obvious step would be to just pay for more powerful test runners.

I see you are using -j2 which seems like a bad idea on underpowered machines especially when we already know your tests are not hermetic.

Shard 2 in particular was unlucky to get a mutant that caused a timeout:

Found 3 mutants to test
ok       Unmutated baseline in 591.2s build + 1489.9s test
Auto-set test timeout to 2h 4m 9s
caught   stackslib/src/net/relay.rs:661:9: replace Relayer::process_new_nakamoto_block -> Result<bool, chainstate_error> with Ok(false) in 617.1s build + 1281.3s test
unviable stackslib/src/clarity_vm/clarity.rs:1262:9: replace ClarityBlockConnection<'a, 'b>::initialize_epoch_2_5 -> Result<Vec<StacksTransactionReceipt>, Error> with Ok(vec![Default::default()]) in 6.8s build
9906.973547644s  INFO timeout after 7449.5s, terminating child process...
TIMEOUT  testnet/stacks-node/src/mockamoto.rs:994:9: replace MockamotoNode::mine_and_stage_block -> Result<u64, ChainstateError> with Ok(0) in 373.9s build + 7449.5s test
3 mutants tested in 2h 45m 6s: 1 caught, 1 unviable, 1 timeouts

Since you tests are already very slow the autoset timeout is very long (2h).

You should probably prioritize investigating the mutant testnet/stacks-node/src/mockamoto.rs:994:9: replace MockamotoNode::mine_and_stage_block -> Result<u64, ChainstateError> with Ok(0) or skip the code that can cause a hang. If you fix or skip the timeouts it will be easier to iterate on everything else.

Additionally, we encountered an issue when using shards where some jobs failed the unmutated tests, while others ran successfully. We attempted to address this by adding the -- -- --test-threads 1 argument to the cargo test command, but, despite this adjustment, the unmutated tests still failed on the job with ID 0.

It is literally just running cargo test in a clean copy of your tree, and it reflects nondeterminacy in your tests. I saw the same thing locally. Your options there seem to be

1. Fix the tests to be hermetic! Seems like a good idea but might be a bit of work. After all, non-hermetic tests might be hiding real bugs.
2. If they are reliable under nextest and that's what you always use, then either send a PR for that or wait a bit for me to add it.