Mismatch Datadog metrics vs Artillery Json report

[Anonycoders]

Hi @hassy 👋🏼

During our tests using Artillery, we noticed that reported metrics (i.e. req/sec) in Datadog are a not matched with those reported by Artillery logs! The numbers displayed in logs are more than those reported in Datadog.
Unfortunately, this deviation caused some issues in our final report.

I hope you can help us with this.

Version info:

2.0.11

Running this command:

run-fargate -o report.json .../test.yml --count 6

In order to find out what is causing the issue, I created a test .yml to generate 300 req/sec for 5 min ( Sustain phase) but each time, the req/sec reported in Datadog is around 150 req/sec but in Artillery logs it shows that it's 300 req/sec!

I tried to decrease the number of arrivals and scale out instead but, even with 6 instances and it still can not reach to 300 req/sec in Datadog.
Also tried maxVusers: 50 in Sustain phase but the result was the same.

Here is the .yml file config for :

config:
  target: "https://xxxxxxxxxx/api"
  http:
    extendedMetrics: true
  phases:
  - name: WarmUp
    duration: 2m
    arrivalRate: 10
    rampTo: 50
  - name: Sustain
    duration: 5m
    arrivalRate: 50
 plugins:
   ensure: {}
   metrics-by-endpoint:
     useOnlyRequestNames: true
     stripQueryString: true
   publish-metrics:
   - type: datadog
      apiKey: ...
      appKey: ...
      prefix: ...
      tags:
        - XXXX
        - ....
      traces:
        ....
 

scenarios:
  - name: List users
     flow:
       - get:
          url: "/users"

I also found this #1569 issue which seems similar.

[InesNi]

Hi @Anonycoders 👋🏻 ,

Thank you for reporting the issue, and for all the info! Will take a look and get back to you.

[Anonycoders]

Hi @InesNi 👋
Do we have any update on this issue?
Please let me know if more information is needed from my end.

[jCOTINEAU]

Hello, we have a similar issue

When running a simple

  phases:
    - duration: 60
      arrivalRate: 1
      name: One per sec

I expect 60 aggregated user and this is the case when running from my machine and with fargate --count 1, but when I bump up the --count I start having weird numbers, is this expected ?

After investigating a bit, if I use --output to look at the json report, the metrics are wrong also in this file, so my guess is that its not coming from datadog.

[hassy]

@jCOTINEAU can you share more on the numbers you're seeing? --count should act as a multiplier, e.g. with the phases config that you shared and --count 2 you should see a total of 120 VUs created. The period may be slightly longer than 60s, but by no more than a few seconds (this is to be expected in a distributed test as Fargate tasks are not guaranteed to start at the exact same time).

[hassy]

and for everyone else seeing mismatched numbers between Artillery and Datadog - can you share the exact config of the chart you're using in Datadog? Sometimes if the correct aggregation is not set in Datadog the numbers can be different to what Artillery is reporting.

[jCOTINEAU]

@hassy hooo that is interesting, i was not understanding that the --count would multiply the phase, I was thinking the phase load would get splited between the count members.

I was expecting the aggregation thingy, will take a deeper look and come back here, thanks a lot for the clarifications.

[hassy]

@jCOTINEAU that's a fair expectation, we need to make it clearer in the docs. We definitely want to add support for distributing the load as you described as well, but no ETA on that yet.

[jCOTINEAU]

Side note, the product is insanely cool, integration with fargate and datadog is super smooth, huge kudos!

[hassy]

@jCOTINEAU thank you, very happy to hear that!

[jCOTINEAU]

ANother question @hassy, you were mentioning datadog aggregation, what value should be setup over there?

[hassy]

It should always be SUM for counter metrics (like vusers.created or vusers.failed) and depending on the time window you might need to add an explicit rollup with a sum as well: https://docs.datadoghq.com/dashboards/functions/rollup/

[jCOTINEAU]

that is interesting, because indeed when i run some more big tests with fargate count=10, the numbers on datadog does not match the one from the json report.

I will dig a bit into the rollup thing, thanks for the clarification.

Edit: damn even when adding a rollup sum from 1 second to X minutes, it does not change anything I am still 2/3 times below the number of vuser created.