Is there a performance profile for buildbot UI

[sananand007]

We want to launch 1000s of builds per day, mostly as PR verification and MRs , wanted to ask if there is any performance profiling done on buildbot hosted UI in terms of scalability and large traffic ?

[tardyp]

Hello,

You can use https://github.com/tardyp/buildbot_profiler in order to look at buildbot performance issue.

If you are planning to build a performance critical bot, we recommand:

• use postgres database and not sqlite
• use multimaster setup, with at least one/two master for builders, and another master for web ui, schedulers, reporters.

If you do that, you should be fine and scale thousands of build a day.

[sphet]

Tardyp,

What is the benefit of the multi-master in this case? I ask because we do many hundreds of builds a day and while postgres has been an improvement we needed, I have not really noticed the need for the multi-master. What symptoms am I ignoring?

[p12tic]

Python is effectively single threaded due to the global interpreter lock and thus only a total of 1 CPU core worth of computations can happen in Python code. In single master case web backend is run in the same process as the build handling and everything else, so each web user reduces the amount of CPU bandwidth available to the rest of the master. If there are many web users this becomes a problem.

[Try2Code]

hi @tardyp!
Does the profiler work with current buildbot release? I get some errors running bbprofiler from the current server base dir.

[Try2Code]

reconfig the server after installing new plugin gives me a 'plugin not found'. do I have to restart the server maybe?

[Try2Code]

sry, my bad: a restart is needed to get the profiler running. works like a charm now. thx for this tool 🙏

[p12tic]

@sananand007

Not sure if this information is still useful to you, but Buildbot easily handles thousands of builds per day and large numbers of historical builds. The web backend can be split off to multiple servers and thus can handle even large amount of user traffic.