[🐛 Bug]: WebdriverIO toMatchSnapshot does not fail tests on mocha test retries

[Winify]

Have you read the Contributing Guidelines on issues?

• I have read the Contributing Guidelines on issues.

WebdriverIO Version

8.36.1

Node.js Version

v20.9.0

Mode

WDIO Testrunner

Which capabilities are you using?

No response

What happened?

When utilizing the toMatchSnapshot expectation in WebdriverIO, it is observed that the test run does not fail as expected when the snapshot does not match the current state of the tested element property.

This behavior is inconsistent with the intended functionality of snapshot testing, where a test should fail if there are discrepancies between the actual output and the stored snapshot.

The test run passes, despite the discrepancy between the snapshot and the current output, with a console log:

Snapshot Summary:
1 snapshot(s) unmatched.

What is your expected behavior?

The test run should fail, indicating that the snapshot does not match the current property.

How to reproduce the bug.

1. Write a WebdriverIO test case that includes the toMatchSnapshot expectation:

describe('Snapshot testing', () => {
  it('should adequately assert texts', async () => {
    await browser.url('https://the-internet.herokuapp.com/');
    await expect($('ul').getProperty('textContent')).toMatchSnapshot('linkList');
  });
});

2. Run the test
3. Change the snapshot text
4. Run the test again

Relevant log output

[chrome 124.0.6367.208 windows #0-0] » \test\specs\snapshot.spec.ts
[chrome 124.0.6367.208 windows #0-0] Snapshot testing
[chrome 124.0.6367.208 windows #0-0]    ✓ should adequately assert texts (1 retries)
[chrome 124.0.6367.208 windows #0-0]
[chrome 124.0.6367.208 windows #0-0] 1 passing (2.2s)


Snapshot Summary:
1 snapshot(s) unmatched.

Code of Conduct

• I agree to follow this project's Code of Conduct

Is there an existing issue for this?

• I have searched the existing issues

[BorisOsipov]

@Winify We need a repository with a reproducible example. Your example and repro steps work well for me.

[0-0] Error in "Snapshot testing.should adequately assert texts"
Error: expect(received).toBe(expected) // Object.is equality

- Expected  - 0
+ Received  + 1

  "
....
    Broken Images
+   Challenging DOM
    Checkboxes
    Context Menu
    Digest Authentication (user and pass: admin)
....
  "
    at Context.<anonymous>
[0-0] FAILED in chrome 

Snapshot Summary:
1 snapshot(s) unmatched.

Spec Files:      0 passed, 1 failed, 1 total (100% completed) in 00:00:05

[wdio-bot]

Thanks for raising this issue 🙏

Unfortunately we can't help you without a reproducible example in this matter. Please read our contributing guidelines on how to create a reproducible example. If you can't provide a reproducible example we will close this issue in 7 days.

If you have any questions please reach out to us on our Discord channel. We are happy to help you out there.

[Winify]

@BorisOsipov Thank you for the swift response,

And I apologize, because as I created a repro example project.... it actually works as expected.
It looks like something in our framework goes awry, but I could not identify what, as I copied our configs to the repro project as well...

All in all, thank you so much

[christian-bromann]

Thanks @Winify , let us know if you are able to reproduce this one, then I am happy to re-open and take a look.

[BorisOsipov]

@Winify There is only one strange thing that I suspect -
should adequately assert texts (1 retries)
(1 retries) - What kind of retries do you use? could you check with and without retries?

[Winify]

@BorisOsipov You are right!

I am using mocha retries:

  mochaOpts: {
    ui: 'bdd',
    timeout: process.env.SPEC_TIMEOUT ?? 90_000,
    compilers: ['ts:ts-node/register'],
    retries: Number(process.env.RETRIES ?? 0)
  },

In the repro project I emitted this config, as I did not think it was neccessary.

If I remove the RETRIES=1 config from the env file, it fails as expected!

[wdio-bot]

Thanks for reporting!

We greatly appreciate any contributions that help resolve the bug. While we understand that active contributors have their own priorities, we kindly request your assistance if you rely on this bug being fixed. We encourage you to take a look at our contribution guidelines or join our friendly Discord development server, where you can ask any questions you may have. Thank you for your support, and cheers!