Question - running multiple load tests in GitHub Actions & running multiple browsers in CI without AWS FARGATE

[Romarionijim]

Hello.

I have two questions

1. I've been using Playwright along with Artillery.
   In UI and API tests - I usually tag my tests and run them via grep ikn the CI e.g npx playwright test --grep @sanity and etc, I usually use a list of greps utilizing strategy matrix.

I was wondering if I can un multiple load tests this way in Artillery - from the documentation your providing an example with only one test file, how can I be able to run multiple tests or tag the tests to run them all instead of specifying more than one workflow?
I was thinking about using a matrix strategy as well to run the files in separate jobs, but I was wondering if there is a built in way?
how I use it for example (without the artillery action).

on:
  workflow_dispatch:

env:
  USER_NAME: ${{ secrets.USER_NAME }}
  PASSWORD: ${{ secrets.PASSWORD }}


jobs:
  load_test:
    timeout-minutes: 60
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: lts/*
      - name: Install dependencies
        run: npm ci
      - name: Install Playwright Browsers
        run: npx playwright install --with-deps
      - name: run load tests
        env:
          BASE_URL: 'https://saucedemo.com'
        run: BASE_URL=${{ env.BASE_URL }} npx artillery run LoginFlowPerfTest.yml
        working-directory: tests/loadTests/artillery

2. If I wanna run a high number of browsers or users, what is the limit if I choose to run them via github actions without using AWS FARGATE? in my company we use GCP and cannot use AWS, i was wondering what is the limit of vusers or browsers that I can spin up and run locally and in GitHub Actions.

Thank you!

[bernardobridge]

Hey 👋

I was wondering if I can un multiple load tests this way in Artillery - from the documentation your providing an example with only one test file, how can I be able to run multiple tests or tag the tests to run them all instead of specifying more than one workflow?

Currently, you can only run one single scenario file. This is in part due to the nature of Load Testing being different than Functional Testing (i.e. they are often not quick tests that you can run one after the other, but rather a long test with a lot of output that needs to be analysed individually). However, it is something we may look to change in the future.

If I wanna run a high number of browsers or users, what is the limit if I choose to run them via github actions without using AWS FARGATE?

We typically do recommend using AWS Fargate because running a lot of browsers at scale is usually a big overhead on the machine running it.

The limit will be up to the CPU/Memory available in the Github Actions runners, and is highly dependant on your test too. You'll have to experiment how much a single runner can handle for your test configuration. I believe Github Actions does have an option for increasing the size of the runners, so you can also try that.

[Romarionijim]

@bernardobridge how much CPU and resources does artillery usually use?
Will there be a support for GCP in the future?

[bernardobridge]

@bernardobridge how much CPU and resources does artillery usually use?

As I said it depends on what your test is doing. You will have to do trial and error to arrive at the right resources in your machine. We don't currently have benchmarks in a single machine for Playwright tests, as our recommendation is to use distributed load testing with AWS Fargate.

Will there be a support for GCP in the future?

We plan to add support for GCP and Azure in the future, but we don't have a timeline yet. We're prioritizing building the best possible experience on AWS first as that's where we see the most interest from our users.

[Romarionijim]

@bernardobridge Thank you for the fast reply.
Because Im investigating the tool and wanna do a PoC, I wanna fully understand running on AWS FARGATE.
If I spin up let's say 3 FARGATE containers, artillery will handle the resource management for me in this case? I wanna understand more than what's written in the documentation if you could elaborate more on this topic if you can.