How to instrument and measure unvisited pages?

[cherewaty]

I'm working on a Next.js app where we've just started adding tests with Cypress. Because there's a lot of pre-existing code, I want to use the coverage reports as a progress report to see how we're progressing... but I'm getting a very limited view from the lcov-report.

Pages in the Next.js app that aren't visited in tests do not appear in the report, so I'm getting inflated coverage numbers. Instead of "what percentage of the total statements in the app are covered?", the report gives me "what percentage of the total statements in pages that the tests visited are covered?"

I took a look at https://github.com/kylemh/next-ts-with-cypress-coverage, which led me to adding the all option to the nyc config. Now pages that haven't been touched show up in the lcov-report... but with all zeros.

I've reproduced at https://github.com/cherewaty/next-and-cypress-example/tree/uncovered-page

uncovered.js should instead show 0/1 statements covered

My hypothesis is that Next.js's dev server isn't even attempting to build pages until they're visited, so babel-plugin-istanbul isn't instrumenting them.

Some paths to explore I've been thinking about:

• Am I missing some configuration parameter that would make this easy?
• Maybe I could hack up the next/babel preset a little bit when NODE_ENV==="test" so all sources get built and instrumented?
• Do I need to run cypress against a "production" artifact from next build instead of the dev server?
• Could I set up a spot where I import everything just for cypress runs?

Happy to make a PR to this example repo if I get this figured out!

[AleksaC]

I'm facing the same issue. I think you are right about next not building pages until they are visited, because it shows 100% coverage for 404 page even though I don't touch it in the tests. According to docs that page is statically generated at build time, hence the coverage.

[bahmutov]

Yeah not sure - but I guess the tests should visit all pages :)

…

Sent from my iPhone

On Dec 19, 2020, at 17:02, Aleksa Cukovic ***@***.***> wrote:  I'm facing the same issue. I think you are right about next not building pages until they are visited, because it shows 100% coverage for 404 page even though I don't touch it in the tests. According to docs that page is statically generated at build time, hence the coverage. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[AleksaC]

Sure, that's the goal. It would just be nice to be reminded that they aren't by failing coverage-check.

[bahmutov]

Can you use the code coverage option “all” with a file pattern to use?

…

Sent from my iPhone

On Dec 19, 2020, at 17:43, Aleksa Cukovic ***@***.***> wrote:  Sure, that's the goal. It would just be nice to be reminded that they aren't by failing coverage-check. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[AleksaC]

Here's how I use it:

{
  "all": true,
  "include": [
    "pages/**/*.js",
    "components/**/*.js",
    "constants/**/*.js",
    "utils/**/*.js"
  ],
  "exclude": ["pages/api/__coverage__.js"]
}

Is this what you meant?

[bahmutov]

Yeah and what happens? It is hard to debug anything like this without seeing the code and trying it

…

Sent from my iPhone

On Dec 19, 2020, at 19:33, Aleksa Cukovic ***@***.***> wrote:  Here's how I use it: { "all": true, "include": [ "pages/**/*.js", "components/**/*.js", "constants/**/*.js", "utils/**/*.js" ], "exclude": ["pages/api/__coverage__.js"] } Is this what you meant? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[AleksaC]

When I run the tests and get coverage report all works well for the pages that are covered by the tests and for the ones that are not I get a result like the one in the image above in the original issue. Also I have a custm 404 page which is also not touched by the tests but the coverage is reported at 100% for it like I mentioned in my first comment.

I can provide a repo reproducing the issue if you want to take a look.

[bahmutov]

Could you provide a repo - not necessarily your private one but make a fresh public one showing the problem. But could you open an issue in cypress-io/code-coverage repo so it does not get lost? And add a link to this issue for full story

…

Sent from my iPhone

On Dec 19, 2020, at 19:48, Aleksa Cukovic ***@***.***> wrote:  When I run the tests and get coverage report all works well for the pages that are covered and for the ones that are not I get a result like the one in the image above in the original issue. I can provide a repo reproducing the issue if you want to take a look. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[AleksaC]

Sure, I was thinking of doing that earlier but I saw this issue so I decided to check here first.