How to get a list of changed files to use in a dynamic trigger?

[ivanov17]

Hello!

I want to use Buildbot to do steps like this:

• Poll a repository with GitPoller
• Run a builder with SingleBranchSheduler if specific branch has changes
• Fetch repository branch and run dynamic trigger on build steps
• Using a dynamic trigger, create a list of top-level directories that have changes, then create appropriately named virtual builders and run a Triggerable scheduler
• Using the Triggerable scheduler, start the virtual builders and perform identical tasks for each directory

I don't want to use multiple repositories, in my case it's more convenient to work with multiple directories.

I have defined a dynamic trigger class and am using it in my configuration. The trigger fires, but I can't get a list of changed files, so no virtual builders are created.

I'm not sure I fully understand the documentation. Maybe I'm trying to get the list of changes in the wrong way.

# SPDX-License-Identifier: Unlicense

class DynamicTrigger(steps.Trigger):

    def getSchedulersAndProperties(self):

        def setVirtualBuilders(changes):

            vbuilders = set()
            for i in (i['files'] for i in changes if 'files' in i):
                dirs = {p[0] for p in (p.split('/') for p in i) if len(p) > 1}
                vbuilders.update(dirs)
            return list(vbuilders)

        changes = self.getProperty('changes') or []
        schedulers_and_properties = list()
        for scheduler in self.schedulerNames:
            for vbuilder in setVirtualBuilders(changes):
                props = self.set_properties.copy()
                props['virtual_builder_name'] = vbuilder
                props['virtual_builder_description'] = str.join(
                    [vbuilder, "virtual builder"], " ")
                props['virtual_builder_tags'] = ['virtual', vbuilder]
                schedulers_and_properties.append(
                    [scheduler, props,
                     scheduler in self.unimportantSchedulerNames])
        return schedulers_and_properties

Or maybe my problem is not in this but in getting changes using GitPoller.

I can see that the changes displayed in the web interface only include merge commits and do not contain file changes, although they should:

Here is also part of my test config file:

# SPDX-License-Identifier: Unlicense

...

c['change_source'] = []

c['change_source'].append(changes.GitPoller(
    repourl=os.environ.get('BUILDBOT_GIT_REPOURL'),
    branches=True, buildPushesWithNoCommits=True,
    category='buildah', pollInterval=60
))

c['schedulers'] = []

c['schedulers'].append(schedulers.SingleBranchScheduler(
    name='buildah-change', reason='Modifying repository files',
    change_filter=util.ChangeFilter(
        category='buildah',
        branch='production'
    ),
    treeStableTimer=3,
    builderNames=['images']
))

c['schedulers'].append(schedulers.Triggerable(
    name="image", reason='Modifying directory files',
    builderNames=['image']
))

imagesFactory = util.BuildFactory()
imagesFactory.addSteps([
    steps.Git(
        name='pull',
        description='Pull repository',
        repourl=util.Property('repository'),
        branch=util.Property('branch'),
        mode='incremental',
        clobberOnFailure=True
    ),
    DynamicTrigger(
        name='trigger',
        schedulerNames=['image'],
        waitForFinish=True,
        updateSourceStamp=True)
])


imageFactory = util.BuildFactory()
imageFactory.addSteps([
    steps.ShellCommand(
        command=["ls", "-l"]
    )
])

c['builders'] = []

c['builders'].append(util.BuilderConfig(
    name='images', factory=imagesFactory,
    workernames=[os.environ.get('BUILDBOT_WORKERNAME')]
))
c['builders'].append(util.BuilderConfig(
    name='image', factory=imageFactory,
    collapseRequests=False,
    workernames=[os.environ.get('BUILDBOT_WORKERNAME')]
))

...

[ivanov17]

I have two updates.

First, I did have problem with the GitPoller configuration. I didn't found this in the Buildbot documentation, but when branches = True is set, the commits that merged into production branch from the main branch are considered to belongs to the main branch. This way, when I filter the branches for scheduler, only merge commits are available on the production branch.

When I set branches = ['production'] for GitPoller, I see in the web interface that all commits merged into the production branch are considered to belong to it. So, if I don't plan to use the main branch, there is no need to fetch changes from all branches.

There is logic in this behavior, but, unfortunately, it is not obvious at first glance.

Second, I still have a problem getting a list of changed files on build steps. I found a solution here that uses Database API, it actually works, but I think there is a simpler solution.

If I try to use .build.allFiles() method on the build step, I always get None. I tried to get a list of the changes with .build.allChanges() method and play around with some scheduler options, but I always get the list contains only the last change. The last change is the merge commit that contains no files, so the list of files in the change is empty.

I see that it is possible to get the list of changes, but for some reason it only contains the last commit. I think I am doing something wrong.

How can I get list of changes containing all changes on the branch?

[ivanov17]

Many thanks to @vibbo, he explained to me what the difference between using DATA API to get all changes for a build and using Build.allFiles() method:

In this context allFiles() means - from all repositories/codebases, but only the most recent Change for each.

#6302 (reply in thread)

So, this is the expected behavior, and if I want to get all changes for current build, I should to use DATA API like this:

@defer.inlineCallbacks
def run(self):
    changes = yield self.master.data.get(('builds', str(self.build.buildid), 'changes'))
    files = set()
    for lst in (i['files'] for i in changes.data if 'files' in i):
        files.update(lst)

    # do something with the resulting list here

or this:

@defer.inlineCallbacks
def run(self):
    changes = yield self.master.db.changes.getChangesForBuild(self.getBuild().buildid)
    files = set()
    for lst in (i['files'] for i in changes if 'files' in i):
        files.update(lst)

    # do something with the resulting list here

Unfortunately, there is no similar examples in the Buildbot configuration documentation. It looks a little strange, because a complete list of changed files is an important thing for building pipelines.

[ivanov17]

Since I'm now using the DATA API to get all changes for current build, my dynamic trigger now looks like this:

class DynamicTrigger(steps.Trigger):
    '''Implements custom trigger step which add virtial builders and tags'''

    @defer.inlineCallbacks
    def run(self):
        '''If the jobs property is not set, sets it from the list of changed
        directories, and then calls the same function of the parent class'''

        changes = yield self.master.data.get(
            ('builds', str(self.build.buildid), 'changes'))
        properties = self.build.getProperties()
        if 'jobs' not in properties or not properties.getProperty('jobs'):
            directories = set()
            for lst in (i['files'] for i in changes.data if 'files' in i):
                for d in (parts[0] for parts
                          in (i.split('/') for i in lst) if len(parts) > 1):
                    directories.add(d)
            self.build.setProperty('jobs', list(directories), 'Trigger')

        res = yield super().run()
        return res

    def getSchedulersAndProperties(self):
        '''Returns the list which contains schedulers for every build target'''

        schedulers_and_properties = list()
        for scheduler in self.schedulerNames:
            for b in self.build.getProperty('jobs'):
                props = self.set_properties.copy()
                props.update(dict(
                    virtual_builder_name=b,
                    virtual_builder_description=f'{b} virtual builder',
                    virtual_builder_tags=['virtual', b],
                    got_revision=self.build.getProperty('got_revision')))
                schedulers_and_properties.append(
                    [scheduler, props,
                     scheduler in self.unimportantSchedulerNames])
        return schedulers_and_properties

Also, I set GitPoller to the same branch as SingleBranchScheduler.

This way I get all the commits that have been merged into the source branch:

c['change_source'] = changes.GitPoller(
    repourl='git://server.name/ops/images.git',
    branches=['production'], pollInterval=60
)

c['schedulers'] = [
    schedulers.SingleBranchScheduler(
        name='change', reason='Modifying repository files',
        change_filter=util.ChangeFilter(
            repository='git://server.name/ops/images.git',
            branch='production'),
        treeStableTimer=10,
        builderNames=['images']
    ),
    schedulers.Triggerable(
        name="buildah", reason='Modifying directory files',
        builderNames=['buildah']
    )
]

So, I mark this thread as solved.