[💡]: Coverage for spawn processes

[cv65kr]

Plugin

None

I have an idea!

There is no possibility to calculate coverage for integration tests.

To write integration test we need to run roadrunner via phpunit e.g. for Temporal plugin - https://github.com/temporalio/sdk-php/tree/master/testing#quick-start, but the same can be happen for GRPC plugin.

So when we run test PS output looks like that: phpunit -> roadrunner -> workers.

root      6555     0  0 15:58 pts/0    00:00:00 bash
root     25163  6555 35 17:45 pts/0    00:00:08  \_ php vendor/bin/phpunit -d memory_limit=512M --no-interaction --debug --verbose --configuration phpunit.xml.dist --coverage-html=coverage-report-phpunit.html --group e2e
root     25168 25163  1 17:45 pts/0    00:00:00      \_ ./temporal-test-server 7233 --enable-time-skipping
root     25173 25163  1 17:45 pts/0    00:00:00      \_ /usr/local/bin/rr serve -c .rr.test.yaml -w tests
root     25184 25173 10 17:45 pts/0    00:00:02          \_ php /src/bin/console app:temporal-worker -p 10 --quiet --no-ansi --no-interaction --env=test --task-queue default
root     25197 25173  8 17:45 pts/0    00:00:01          \_ php /src/bin/console app:temporal-worker -p 10 --quiet --no-ansi --no-interaction --env=test --task-queue default

I tried pcov and xdebug and result is the same - code is not covered.

Generally I don't have idea how solve this problem, so I will raise it here to open discussion.
Maybe new plugin which generate coverage files for spawned workers? Because it's relatively easy to merge coverage files. Maybe new phpunit driver?

[rustatian]

Hey @cv65kr 👋🏻
I transferred the ticket into the discussion.

Maybe new plugin which generate coverage files for spawned workers?

I guess this is a good idea, but this is hard for me to help here since I never run a phpunit and have no clue how this thing works 😭

Might be @butschster / @wolfy-j might give the advice here 💯

[acelaya]

I would also be interested in supporting some way to collect code coverage in RoadRunner, for integration/E2E tests.

I have only tried using pcov sebastianbergmann/php-code-coverage#1003, but I understand by the original post that xdebug wouldn't work either

[rustatian]

Hey @acelaya 👋 Thanks for working on that 👍. If you need any help from me, feel free to ping me on our Discord or here 😃

[acelaya]

Ok, I finally found the root cause. It was related with pcov.directory, but not exactly.

The problem seems to be somehow related with the fact that I have my .rr.yml config files in a subdirectory (to keep things tidy, as the project root is already full of other config files), which affects some relative paths defined in it, like server.command.

If I move that config file to the project root directory and adjust those relative paths, then code coverage with pcov works as expected, without changing anything else.

[rustatian]

Great job 👍
Could you please write here a small instruction what to do in such a case? I mean, when a user needs to generate code coverage. Where should be .rr.yaml, what should be in the config, might be how to start RR (might be use -w to set a working directory).
I'll put this in the documentation.

[acelaya]

I actually didn't think about the -w flag!

I suppose the working directory is inferred from the rr.yaml config file location, if not specified. What I did was moving that file to the project root.

However, I have just tried moving the file to its original subfolder, and specified the working directory via -w, and that also worked, with the only implication that routes inside the rr.yaml file needed to be adjusted to be relative to the project root, not the file location.

So I guess you can document those two approaches.

[rustatian]

Yeah, definitely :) But, I'm not a PHP dev (I mean, at all) 😆
It would be super cool if you'd be able to write a short instruction here about I wrote previously. And I'll put it in the proper docs section 😃

[acelaya]

I guess the explanation depends on what is intended to be documented. When I first started participating in this discussion, I was having some problems generating code coverage for E2E tests, where the test runner is executed on a different process than the code under test.

Before using RoadRunner, I was using a different app server (openswoole), and for that one, I came up with a workaround which required to conditionally and programmatically call the appropriate logic to collect and generate code coverage reports, which works out of the box for unit tests where the code under tests is imported in the same process executed by the test runner.

This logic is a bit cumbersome, and has some implications, like the fact that you need to make sure there's only one worker serving HTTP requests, otherwise you end-up again with multiple processes loading your source code, and you would need to coordinate them somehow to properly generate the coverage reports. Also, logic running in job workers would never be marked as covered.

When I moved to RoadRunner, I assumed this approach would work as well, but my surprise was that I was getting reports with 0% of coverage.

After googling a bit I ended up here, and incorrectly assumed there was an actual known issue with RoadRunner and coverage reporting tools, but it turns out the original author of this discussion, was actually suggesting something else, implementing something in RoadRunner itself, so that the server workers could report code coverage and let the test runner on a different process consume it and generate coverage reports transparently as you would do with unit tests.

This approach is obviously much better, but considering I already had the other one working, I thought it would be easier to get that one working in the short term. Also, it has the benefit of being agnostic to the app server.

With the context clear, it turned out the reason I was getting a 0% coverage was because of the incorrect setting of the working directory, which could be solved with any of the two approaches described here https://github.com/orgs/roadrunner-server/discussions/1440#discussioncomment-8486432

It's also worth mentioning I only tried with pcov (one of the two coverage drivers supported by PHPUnit, the most popular PHP test runner/test framework). XDebug, the other coverage driver, might behave differently.

[rustatian]

Cool, thanks for the explanation 👍

[cv65kr]

Long story short:

If you want to have a integration test, on the beginning of test you need to run roadrunner (and this is separate process).
So now your top looks like that:

• phpunit (Main process)
• roadrunner (2 workers)
• • php worker
• • php worker.

According to way how coverage is calculated in phpunit - https://github.com/sebastianbergmann/php-code-coverage/blob/main/src/Driver/PcovDriver.php the test block looks like

START test
- pcov/xdebug START
- detached RR
- test
- assertions
- pcov/xdebug START
END test

So in general main process in this case phpunit is not able to calculate and sum coverage from detached php workers.

Actual behaviour: coverage is calculated without php workers

---

So @acelaya the idea to add pcov start/end in worker is good start, but it's kinda bad solution because you need to have production and testing workers - so you need to maintain two things.

The problem is still the same, phpunit as a main process still is not able to calculate coverage from detached processes.

Ah, gotcha. I guess pcov is mistakenly using stdout to print its data. Wrongly, because generally all logs, all informational stuff should be sent to stderr.

Is it possible to redirect (or use some pcov option) to send the data through stderr?

pcov/xdebug functions not printing anything to stdout:
https://github.com/sebastianbergmann/php-code-coverage/blob/main/src/Driver/XdebugDriver.php#L110
https://github.com/sebastianbergmann/php-code-coverage/blob/main/src/Driver/PcovDriver.php#L50
https://github.com/krakjoe/pcov

[cv65kr]

I checked as fast poc - storing all coverage outputs (from php workers) to files and merge it in the main process. Generally looks good.

Like I said the ugly thing is that we need to keep two workers files for testing/prod, but this solution is possible to do even without RR.

I will think about single entrypoint and will try to back to you asap.

[wolfy-j]

May I ask why not calculate coverage within tests without the use of RR? All drivers (HTTP, GRPC and Queue) are properly isolated from RR instance, so calling GRPC service via stub server will fully emulate the RR behaviour. You can even reuse it between tests to check for memory leaks.

[wolfy-j]

We don't have stub server for queues (we can build one if really needed), but for GRPC you can test using Invoker - https://github.com/roadrunner-php/grpc/blob/3.x/src/Invoker.php

From PHP standpoint there are no other magic going on, it can be executed in a separate CLI and should produce response similar to RR.

[cv65kr]

What about temporal sdk? which uses temporal test server and you are not able to isolate it

[wolfy-j]

Yes, Temporal will be an exception, unlike all other drivers it uses async pipeline which is almost impossible to decouple from code (without reinventing temporal in PHP). There a internal trick we can use to replay history of communication between RR and worker, but it's not exposed to user level. So for Temporal you might still need to use external coverage instrument, I have not worked with them in such regard so it's hard for me to advice. I can only say that RR processes behave similar to simple PHP CLI, so if you can do coverage of CLI you can do coverage of any RR worker.