Am having the same problem here. The sub_artifact() method just doesn't seem to do what I want it to do.

My use case: I want to generate a bunch of Creative Commons logos into the build folder. How many files I need to generate I can't say beforehand.

My problems:

• These files do not have a corresponding file below the project folder (except when a source checkout of the plugin is used).
• As there are multiple files, I can't resort to the trick of declaring the plugin config file as the source file of the artifacts.
• If I don't declare any sources for the artifact, the prune stage will just remove it again

So, what to do? I'm currently trying to use this code (which results in the generated icon getting pruned)

    def icon_path(self, license):
        icon_target_path = (
            '/static/lektor-creative-commons/{type}/{version}/{size}.png'
        ).format(**license)
        icon_source_path = os.path.join(
            os.path.dirname(__file__), 'assets', license['type'],
            license['version'], license['size'] + '.png'
        )
        ctx = get_ctx()
        # ctx.record_dependency(icon_target_path)
        @ctx.sub_artifact(
            icon_target_path,
            sources=[],
            # sources=[ctx.source.source_filename],
            source_obj=icon_source_path
        )
        def copy_icon(artifact):
            artifact.replace_with_file(icon_source_path, copy=True)

Using the currently commented out sources=[ctx.source.source_filename], leads to the artifact being created for every page that includes it. That works, but slows down the build significantly and spams the build logs as every second line is that artifact.

So how am I supposed to use ´sub_artifact()`, or more to the point of this bug report, how to create a single arbitrary file using the build system?

Thinking further about this: a possible workaround may be to copy the file first into the assets folder to a known location inside the project by hand, and then create an asset that depends on that file.

That would mean that I have to keep track by hand wether the file in /assets is up to date, but at least then I can use the build system afterwards in what I think is correct.

Still that seems like a pain in the ass, and also burdens the user with understanding that there are certain files in his assets folder that he is not allowed to touch. However if that is the only way to do it¿

@nixjdm Recommended to look into VirtualSource objects like these https://github.com/nixjdm/lektor-tags - not sure what to make of these yet, but I will take a look.

+1, this should be documented.

For the "just write one file" case I think lektor-atom is a somewhat simpler example, see

    def on_setup_env(self, **extra):
        self.env.add_build_program(AtomFeedSource, AtomFeedBuilderProgram)

in https://github.com/lektor/lektor-atom/blob/master/lektor_atom.py

Edit: note that the simple case is documented under https://www.getlektor.com/docs/api/environment/add-build-program/. A more complex example could maybe be added to guides.

I would like to continue work on this. It seems that both the CC-Plugin as well as lektor-atom are bitten by the same problem, that their generated files are removed in the prune stage.

A bit of debugging on the cc-plugin seems to suggest that the problem may be with the dependency recorder, which doesn't seem to like to record the dependencies that are not inside the lektor project folder. So for example with this code:

ctx = get_ctx()
@ctx.sub_artifact(icon_target_path, sources=[])
def copy_icon(artifact):
    ctx = get_ctx()
    ctx.record_dependency(__file__)
    ctx.record_dependency(icon_source_path)
    artifact.replace_with_file(icon_source_path, copy=True)

I would expect the resulting file to have a dependency on the source code of the plugin file, as well as a dependency on the icon file itself.

As long as the plugin is installed as a package this would work fine, but break as soon as the plugin is installed aas a project dependency with a ValueError like this:

ValueError: The given value ('/Users/dwt/Code/Projekte/homepage/packages/creative-commons/lektor_creative_commons/assets/by-sa/4.0/88x31.png') is not below the source folder ('/Users/dwt/Code/Projekte/homepage/test')

Sow how is this supposed to work? The Documentation recommends that this approach should work:

        ctx = get_ctx()
        @ctx.sub_artifact(icon_target_path, sources=[])
        def copy_icon(artifact):            
            artifact.replace_with_file(icon_source_path, copy=True)

But that just means that the generated file will be removed in the prune phase. Adding the dependencies as noted above fails with a ValueError.

Also this way of writing it explodes with the same ValueError.

        ctx = get_ctx()
        @ctx.sub_artifact(icon_target_path, sources=[
            icon_source_path,
            __file__,
        ])
        def copy_icon(artifact):
            artifact.replace_with_file(icon_source_path, copy=True)

I haven't looked into the details yet, but it seems correct to me, that plugins should be able to create files that depend on files that are distributed as part of the plugin source code.

Is there a reason why this is bad? If not I would recommend allowing the paths where plugins are installed as valid dependency paths to make authoring correct plugins easier.

This may not all be completely correct, but...

Two types of sources (dependencies) are recorded for artifacts: primary, and secondary.

The artifact will be pruned when there are no primary sources (or those that are listed no longer exist.) The point of the pruning process is to provide a way to remove stale artifacts from the build tree. E.g. when a page or attachment is deleted, we want to try to ensure the corresponding file gets removed from the build tree.

Primary sources are, I think, those which are passed in the sources parameter of ctx.sub_artifact (or BuildProgram.declare_artifact, or BuildState.new_artifact).

As you note, sources must live within the project tree. I don't really know what the original reasoning is behind this. I suspect the idea is to make sure sources are part of the site source: i.e. those things which would reside within the site's git repo. External python distributions, to me, fall outside of this scope.

So, in your case, to prevent pruning of your artifact, it must list a primary source within the project tree. I don't completely understand your use case, so I can't suggest what the best choice would be, but some options are:

• the page source (or template source) for whatever page it is that triggered the production of the artifact in question
• the project config .lektorproject file
• the front page source content/contents.lr

If there is some source file that is responsible for "triggering" the production of the artifact in question, then listing that as the primary source is probably the best strategy. That way if that source goes away, the artifact will (hopefully) be pruned.

As for ensuring the artifact is rebuilt when a source outside the source tree changes, here are a couple of options.

If the source is in an external distribution whose version is pinned in the project's .lektorproject file, then declaring a dependency on that config file, will handle the case of distribution updates. That's probably the simplest option.

Another option is to declare a dependency on a virtual source object registered by your plugin. The virtual source should override VirtualSourceObject.get_checksum and/or VirtualSourceObject.get_mtime. Those will be used to determine when the virtual source "changes" — when it does the artifact will be rebuilt.

That's some good comments. Before I delve deeper (probably not before Thursday) some remarks about things I've tried:

• I have tried making the pages that trigger the creation of the asset a primary source - but that leads to the asset being rebuilt for every page it is used on, doubling my build time. Now that I think about it - maybe providing the checksum could prevent the rebuild? That would be a really nice way to fix this.
• Virtual Source Objects: I have had no success getting that to work in a way that wouldn't be pruned anyway. But I can give it another shot.

Getting this thought further: If Virtual-Source-Objects are the right way to go, it could be a really good option to make it damn simple for plugins to get their own plugin virtual source object generated for them by default for lektor to make it dead simple to create files for it, that don't have any corresponding to real objects.

For the use case, I'm struggling with both the Atom Feed plugins lektor/lektor-atom#10 as well as resolving the same issue with the CC-Icons plugin humrochagf/lektor-creative-commons#3

• I have tried making the pages that trigger the creation of the asset a primary source - but that leads to the asset being rebuilt for every page it is used on, doubling my build time. Now that I think about it - maybe providing the checksum could prevent the rebuild? That would be a really nice way to fix this.

Any time any dependency is changed (edited) that would result in a rebuild. So if a number of pages are listed as dependencies, changes to any of those pages will trigger a rebuild. Also, my guess is that if you generate the same file as a sub-artifact from multiple pages, and the dependency information is not clean (in such a way that it always appears stale), it will be built multiple times during a single site build.

• Virtual Source Objects: I have had no success getting that to work in a way that wouldn't be pruned anyway. But I can give it another shot.

I don't think (though I'm not sure) that virtual source objects work as primary sources — at least not to the extent that they would prevent pruning. A primary source has to correspond to a real on-disk file to prevent prunage.

For the use case, I'm struggling with both the Atom Feed plugins lektor/lektor-atom#10 as well as resolving the same issue with the CC-Icons plugin humrochagf/lektor-creative-commons#3

From a first glance, lektor/lektor-atom#10 looks like a different issue. It seems to be triggered by having (or enabling) translated pages (alternatives). At first guess, that seems likely to be a bug in Lektor's dependency checking code whereby pages appear missing during the dependency check unless the alt-specific overlay file exists (or something like that.)

I'll try to look at this when I get the chance. (It probably won't be until after next weekend.)

@dwt I've just looked at lektor/lektor-atom#10. That turns out to be a related but not quite the same issue. (See lektor/lektor-atom#36)