snapshot testing

[chenrui333]

Any helper or tools that you can think of or recommend to help with snapshot testing? Thanks!!

[jcamiel]

Hi @chenrui333 ,

One way to test a snaphot version would be to run Hurl's integration suite tests. These tests are run for each commit on the GitHub CI/CD but can also be easily locally executed.

Your can find these tests under integration/:

• integration/tests_ok
• integration/tests_failed
• integration/tests_error_parser
• integration/tests_error_lint

Each integration test consists of:

• a .hurl file: this is the Hurl file that we want to test
• a .py file: this is a Flask endpoint that is used by the tested Hurl file
• a .options file: list of command line options to execute the tested Hurl file
• a .out file : the expected stdout for this Hurl file (with a variant .out.pattern when there is some variant like date etc...)
• a .curl file: the expected curl command greped from the verbose run of the tested Hurl file.
• a .json file: the expected JSON export for the tested Hurl file
• a .html file: the expected HTML rendered Hurl file
• a .err file: the expected sterr for this Hurl file (with a variant .err.pattern when there is some variant like date etc...)
• a .exit file: the expected status code

Some files are optional (tests under tests_ok have no .exit file for instance).

To run and test the integration suite, you will need Python 3.9+. The simplest way is to create a Python virtual environment:

$ python3 -m venv .venv
$ source .venv/bin/activate

Then start the test local servers for the test suite:

$ ./bin/test/test_prerequisites.sh

Once the script is launched, you can execute the test suite:

$ cd integration
$ python3 integration.py

This script will run and tests all integration tests. The hurl binary which is used depends on your $PATH variable. For instance, if you want to do some modification to the code and test it, you can export your PATH:

$ export PATH=/Users/jc/Documents/Dev/hurl/target/debug:$PATH

and rerun the tests suite:

$ cd integration
$ python3 integration.py

Of course, you also have units tests that can be executed like this:

$ cargo test

Just keep in mind that there is currently some work to streamline the current integration workflow but the idea will still be to have a big suite of Hurl files and to test those extensively.

[chenrui333]

Sorry, I was not clear, this is not for hurl project, but for the project which is using hurl. :)

[jcamiel]

☺️My bad! Could you expand a bit your question?

Meanwhile, I can give you some information about how we use Hurl in our company. We are a 30 developper team working on Spring applications + VueJS. We have ~200 Hurl files, hand written and commited in our repos (tests go from one or two requests to 100s of requests). We use GitLab CI/CD to build and check our applications and build a docker image. Inside GitLab CI/CD, we instanciate a container with our applications, and run our Hurl file just with hurl --test *.hurl. We construct a HTML report with --report-html that is published on GitLab pages. When we run our application, we mock any web service that can be used with WireMock. The test suites is executed is ~10 minutes. We have this setup for more than 3 years and are extremely happy with it (very fast in a CI/CD, can be locally run if needed, no false positive or flaky test)...

[chenrui333]

I am mostly think like jest snapshot testing so that I can capture the tests as like a dump kind of thing, it would be easy for auditing as well, let me know if that makes sense. :)

[jcamiel]

I don't know Jest snapshot testing, but we're thinking about creating browsers plugins to capture a web session in a Hurl file. Just an idea for the moment...

[chenrui333]

yeah, that is a good idea!! But didn't playwright already have such process? https://playwright.dev/docs/codegen

[jcamiel]

Yes, it does! We don't try to compete with this giant and are focused on a simple CLI tool. But we can take inspiration from the best 😇

[chenrui333]

I am all for it :) Happy to switch my tests off playwright LOL

[chenrui333]

by the way, I am starting to do the similar things like you did over the last three years, do you have practice guide against different environments?