Multimaster setup with UI and per-project buildmasters

[an-ky]

Hi,

I'm currently setting up a buildbot using a multimaster configuration with a dedicated master for UI and several masters each responsible for a dedicated project. On those project masters all relevant objects (shedulers, reporters, builders) are configured. Builds are carried out on KubernetesLatentWorker, where each master has a dedicated set available.
Changes are pushed from Gitea through webhooks.
My initial idea was to route every change through the UI master which will spread the change through wamp (crossbars) to inform the project masters about the change so they can handle it. Apparently this either does not work or I missed some configuration because the UI master received the change and queues it (2021-10-10 19:41:35+0000 [-] injected change nn showing up in the logs) but then nothing more happens. If I setup the webhook to directly address the project-specific master, everything works fine.

Is the initial setup idea possible at all? If not, is there a chance to change the setup to enable this behavior?
I'd happily try to provide a PR if this needs changing buildbot's message dispatching behavior but some tips about where and how to start would be quite appreciated.

My buildmaster config looks like this by default (aka exactly like this for the UI master. Builders, schedulers and workers are created through a YAML config for the project masters)

config = {
            "buildbotNetUsageData": None,
            "buildbotURL": os.environ.get("TYR_URL"),
            "title": os.environ.get("TYR_TITLE", "Tyr"),
            "db": {
                "db_url": os.environ.get("TYR_DB_URL"),
            },
            "protocols": {"pb": {"port": 9989}},
            "www": {
                "port": 8010,
                "plugins": {
                    "waterfall_view": True,
                    "console_view": True,
                    "grid_view": True,
                    "badges": {},
                },
                "allowed_origins": [
                    "https://*.sub.example.com",
                    "https://*.example.net",
                ],
                "change_hook_dialects": {
                    "gitea": {
                        "secret": gitea_secret,
                    }
                },
            },
            "services": [
                reporters.GiteaStatusPush(
                    os.environ.get("TYR_GITEA_URL"),
                    token=util.Secret("tyr/gitea-token"),
                    verbose=True,
                )
            ],
            "builders": [],
            "change_source": [],
            "schedulers": [],
            "workers": workers,
            "multiMaster": True,
            "mq": {
                "type": "wamp",
                "router_url": "ws://crossbar.services.svc.cluster.local:8080/ws",
                "realm": "realm1",
                "wamp_debug_level": "error",
            },
            "secretsProviders": [
                secrets.HashiCorpVaultSecretProvider(
                    vaultToken=os.environ.get("TYR_VAULT_TOKEN"),
                    vaultServer=os.environ.get("TYR_VAULT_SERVER"),
                    secretsmount="kv",
                    apiVersion=2,
                )
            ],
        }

[tardyp]

From what I read, your implementation looks good. It is supposed to work like this.

Changes are supposed to be received by scheduler in this method.
https://github.com/buildbot/buildbot/blob/master/master/buildbot/schedulers/base.py#L189
maybe you can put some debug logs there.

I have doubts however this setup is deployed in a lot of instances. Very few people are deploying multi-master.

I usually suggest people to put schedulers in the UI master. the ClusterService is known not to work very well, i.e. election of which scheduler is active when a scheduler is configured on several master do not work very well.

So maybe this is why you are the first to see this issue.

[an-ky]

So, well after adding some log statements and triggering the webhook everything worked as expected. I wonder whether it did not work yesterday due to some deployment errors (broken service in the kubernetes cluster etc. pp) but for now I'll look how stable it runs and how good it will scale when adding more project masters.

Thanks for the hint to the scheduler, getting deeper insights into the codebase and how everythink collaborates is nice.

Best regards
an