How do I get the list of files from change_source?

[jhagg]

I want to trigger a build using 'sendchange'. That command has a few options and then a list of files as arguments.
This list of files should then be used as part of a shellcommand.

I guess I should use Interpolate somehow, but I can't find the complete list of properties, only examples of how I would use util.Interpolate().
Yes, Interpolate works, I managed to get prop:buildername. :-)

But where do I get the property list and list of files?
Any help is welcome!

[vibbo]

This is a Renderer, that will return list of files from changes (for default codebase "")

from buildbot.plugins import util
from twisted.internet import defer

@util.renderer
@defer.inlineCallbacks
def r_files(props):
    changes = (yield props.master.db.changes.getChangesForBuild(props.getBuild().buildid))
    files = set()
    for change in changes:
        if change["codebase"] == "": #  Presumably you want to filter by codebase
            files.update(change["files"])

    #  You can return what you need here
    return list(files)

[jhagg]

Maybe I'm doing something wrong, but I can't seem to get it to work.
I have this line:
fact.addStep(steps.SetProperty(property='in_files', value=r_files))
But the result is always empty.

By the way, it seems strange that there isn't some property already available, or is 'files' something people normally
don't use? :-)

[jhagg]

Hmm, it seems as if branch name is required. When I add '-b something' to sendchange I get my list. Strange.

My current sendchange line:

buildbot sendchange -m HOST:PORT -b SOMETHING -a USER:PWD -W [email protected] -C default filename1 filename2 ...

Well, as long as it works. :-)

[ivanov17]

Yes, it works for me, but I can't figure out why changed files are available from the database API and not available from the build itself with self.build.allFiles(). @vibbo, could you explain this in a nutshell?

[vibbo]

Hi.

I will try. Just note, that my knowledge is also limited.

AFAIK this all boils down to Changes vs. Sourcestamps. Build is started for one sourcestamp per repository/codebase (usually one repository/codebase ). It is a single commit from that repository. You can inspect this in Debug tab of given build.

• If this is a merge commit there are no files associated with that commit.
• Or it the build was started only for the most recent commit of tracked branch (due to use of a Collapse Request Functions) there are only the file changes for that commit.

This is how self.build.allFiles() collects Changes. In this context allFiles() means - from all repositories/codebases, but only the most recent Change for each.

This is why IMHO listing the changes via DATA API getChangesForBuild is more convenient as it gets all Changes since the last successful build, thus giving you more complete list of changed files you can work with.

Best regards,
vibbo

[ivanov17]

@vibbo, thank you very much, that's a great explanation.

I tried to play with dynamic trigger configuration and use an alternative method to get the list of the changed files using the Data API. It seems it works as well as your code. At least I haven't had any problems with it yet.

# SPDX-License-Identifier: Unlicense

@defer.inlineCallbacks
def run(self):
    changes = yield self.master.data.get(('builds', str(self.build.buildid), 'changes'))
    properties = self.build.getProperties()

    # here I check current properties and get directory names from changes  
    if 'jobs' not in properties or not properties.getProperty('jobs'):
        directories = set()
        for lst in (i['files'] for i in changes.data if 'files' in i):
            for d in (parts[0] for parts in (i.split('/') for i in lst) if len(parts) > 1):
                directories.add(d)
        self.build.setProperty('jobs', list(directories), 'Trigger')
    res = yield super().run()
    return res

If anyone wants to use this code, note that I'm using data for iteration from changes.data, not directly from changes.

This is because the changes are not returned as a list, but as ListResult class that based on UserList from the collections module. See here for details:

buildbot/master/buildbot/data/base.py

Line 208 in 968c203

class ListResult(UserList):