Cannot use Interpolate in ChangeFilter

[angeloudy]

I recently upgrade buildbot to 3.11 and everything stopped working.
The error I'm getting is as below:

2024-05-01 22:37:41+1000 [-] Loading configuration from '/var/buildbot/master.cfg'
2024-05-01 22:37:41+1000 [-] error while parsing config file:
	Traceback (most recent call last):
	  ...
	  File "/var/buildbot/master_lib.py", line 555, in create_job
	    change_filter=util.ChangeFilter(category=None,
	  File "/usr/local/lib/python3.8/site-packages/buildbot/changes/filter.py", line 86, in __init__
	    self.filters += _create_filters(
	  File "/usr/local/lib/python3.8/site-packages/buildbot/util/ssfilter.py", line 174, in _create_filters
	    values = extract_filter_values(eq, prop + '_eq')
	  File "/usr/local/lib/python3.8/site-packages/buildbot/util/ssfilter.py", line 26, in extract_filter_values
	    raise ValueError(f"Values of filter {filter_name} must be list of strings or a string")
	builtins.ValueError: Values of filter repository_eq must be list of strings or a string

The line of code that had issue is

def create_job(project, build_steps, path='', github=None, watch_path=None, workers=None, builder_properties=None, is_test=False, jira_project=None, add_build_number=False):
    c['schedulers'].append(schedulers.SingleBranchScheduler(
            name=project,
            change_filter=util.ChangeFilter(category=None,
                                            repository=util.Interpolate('https://github.com/%(kw:repo)s', repo=github),
                                            branch='main'),
            builderNames=[project],
        ))

So, the issue is I'm using a method create_job to dynamically create my builders, and I used Interpolate to dynamically get the project name. However, in the new version it's expecting the repository to be either a string or a list of strings.

How do I fix it?

[tdesveaux]

What's the type of your github arg in create_job? Would a simple Python format work here?

repository=f'https://github.com/{github}'

[angeloudy]

@tdesveaux Yes, that can do the trick, but I have many of them and some of them are more complicated, for example

util.Interpolate('https://username:%(secret:github)[email protected]/%(kw:repo)s', repo=github)

[tdesveaux]

You could use a regex in these cases with repository_re arg.

repository_re=re.compile(fr"https://username:.+@github\.com/{re.escape(github)}")

If you have other cases that are more complicated, you can even use a function with repository_fn (ChangeFilter doc)

(Side note, but you might be interested by PR #7543 as it would allow you to not set the secret in the URL in the future).

PS: not weighting on the question of whether ChangeFilter should take renderable values, just trying to provide a solution that would work with the current release.

[p12tic]

Introduced in #6630.

[p12tic]

I have looked into this problem and unfortunately it would be complicated to re-add support of renderables to ChangeFilter and the benefit of doing so is small.

Using renderable with ChangeFilter is a roundabout way of using the values directly, because the values will need to be rendered on master startup anyway. The only exception is secrets, because they come from a different source. However, if secrets are used in ChangeFilter, then the secret values are already present in various places, such as in the database and Buildbot logs. Therefore there's very little reduction of security if the secrets are specified directly in the Buildbot configuration file.

I think the suggestions by @tdesveaux make sense, in particular the regex option. With that Buildbot will still perform the same filtering as before, but secret will not need to be provided at all.

I have opened #7557 to improve documentation of the current behavior and the best practice.

[p12tic]

I think #7557 addressed this bug, please reopen if you disagree. Thanks!