[scalability] a script to automate threshold estimations

[mimowo]

What would you like to be added:

A script to automate the estimation for scalability thresholds, based on the experiment here: #2066 (comment).

In order to make the estimations more reliable we should only use main branch. For that, it would be good to setup the periodic build of the main branch for perf tests.

Note: Once the script is added, and we have data for the main branch, use it to set new values.

Why is this needed:

There is no easy and reliable way to set the thresholds, it leads to flakes, and unnecessary chore.

Having an easy way to collect the data per build will also enable us to see visually if there is a drop in performance on the main branch, which isn't picked by the current automation.

Also, we could use it to confirm our wins when we optimize the performance, by visualizing easily that there
is better performance based on historical main builds.

[mimowo]

/cc @alculquicondor @trasc

[alculquicondor]

I don't think we should be changing the values that often.
I prefer we don't have to maintain yet another script.

[mimowo]

Right, there would be some maintenance cost, which makes me hesitant as well.

The idea is that currently is there is a failure it is not clear it would be not clear to me how much to bump the thresholds, and collecting the data manually takes time.

All In all, I'm ok to park it for now and see how it plays out, and invest only if bumping becomes to much chore.