Ergonomics around Asset Processing

[inodentry]

It is common to want to do some processing on an asset as soon as it is loaded, to prepare it for use.

For example, setting texture sampling parameters (relevant discussion).

Currently the way to do that is by having a system that runs every frame (possibly confined in a state), either waiting for an AssetEvent, or just polling/checking the handle. This is very unwieldy and a lot of boilerplate. I'd like it to be more ergonomic.

It would be nice to be able to simply register a system to be automatically run once, when the asset is done loading. Ideally, it should be able to get the asset/handle directly as an input.

Here is what such a system might look like:

fn texture_processor(
    In(handle): In<Handle<Texture>>,
    mut assets: ResMut<Assets<Texture>>,
    // ... any other system params
) {
    let texture = assets.get_mut(handle).unwrap();
    // ^ unwrap is ok, because we know the system will run after the asset is available
    
    // TODO: do things with `texture`, knowing that this system will only run once
}

To simplify this further, the asset itself could be provided directly, to avoid the need for the assets resource, get_mut, and unwrap() (as these would be practically necessary boilerplate, probably to be found in every such processing system):

fn texture_processor(In((texture, handle)): In<(Texture, Handle<Texture>)>) {
    // do something with `texture`; the `handle` is available too if we need it
}

If we support this, then, for the sake of ergonomics, all combinations of asset and handle inputs could be supported:

• (T, Handle<T>)
• (Handle<T>, T)
• Handle<T>
• T

The above could all be valid input types for systems accepted as asset processors. It would let the user pick what they need.

Registration

Ideally, I'd like it to be possible to register systems to run only on specific assets, not necessarily always on all assets of a given type.

For example, it could be done by registering the processing system with the AssetLoader when calling load:

fn setup(server: Res<AssetServer>) {
    let handle = server.load("my_texture.png").unwrap();
    
    // texture_processor system will be run when this asset gets loaded
    server.set_processor(handle, texture_processor.system());
    
    // also, register a processor for all textures
    // (maybe this should be done in the App builder instead?)
    server.set_default_processor::<Texture>(default_texture_processor.system());
    
    // i'd expect the default processor (if any) to run first
    // followed by the specific one for this handle (if any)
}

I don't know if registering systems on the AssetServer like this (or in general outside of the App builder) can even be feasibly supported; I am just illustrating what I'd ideally like the UX to be.

Adding the default processor is something that could be done from the App builder.

However, I don't think registering a system only for specific asset handles is something that can be done from the App builder, but it is a valuable use case to be supported, so we need to figure something out for that.

I have no idea what the implementation of this feature would be.

• How would it interact with bevy's scheduling, to actually run these systems as intended?
• How would the asset/handle be passed into the system? That probably needs some plumbing...
  • The system In parameter syntax seems like such a natural fit for this...
• Would we support adding multiple processor systems? If yes, do they get internally represented as a stage? system set? What about order of execution?

Discussion welcome!

[zicklag]

I haven't looked super in-depth to this proposal and I don't have time to weigh in on this properly, but I definitely 👍 the general idea. I have run into a similar issue: I have a custom Bevy map loader that loads a texture for the tilemap and it needs to set the filtering mode for the texture, but as it stands, it's quite awkward to set that filter because I have to wait for the asset to load in some system before modifying the filtering mode.

Like, maybe we could pass in a closure that gets run immediately on the asset after the asset is loaded by the corresponding AssetLoader?

let texture_handle = asset_server.load("face.png").map(|x| x.sampler = SamplerDescriptor {
     min_filter: FilterMode::Nearest,
     mag_filter: FilterMode::Nearest,
     ..Default::default()
});

This way there's no scheduler interaction or anything and it is really obvious what is happening. Again, didn't spend much time thinking about this, but there's some food for thought. 🙂

[inodentry]

Your closure proposal is much more limited in what use cases it enables, because it can only have access to the asset itself (and perhaps some variables moved into it from the environment). A system can take any additional SystemParams, such as resources and queries (like any bevy system), making it a lot more powerful (and idiomatic to bevy).

[zicklag]

That's true, I didn't think about needing access to other resources to do the necessary modification. From my use-case I'm immagining that I have the info about how I want the asset loaded when I issue the request to load it.

[cart]

Yeah handling this nicely seems valuable, but I want to consider "generic" solutions to the problem before trying to build something custom for assets. Every new concept we add incurs cognitive load.

Ex: rather than adding "asset stages" or AssetServer-owned systems, we could instead add "event systems":

app.add_system(handler.event_system())

fn handler(In(event): In<AssetEvent<Mesh>>, mut meshes: ResMut<Assets<Mesh>>) {
  if let AssetEvent::Created { handle } = event {
    // do_thing
  }
}

However I also think that the new EventReader param almost removes the need for "event systems":

app.add_system(handler.system())

fn handler(mut events: EventReader<AssetEvent<Mesh>>, mut meshes: ResMut<Assets<Mesh>>) {
  for event in events.iter() {
     if let AssetEvent::Created { handle } = event {
        // do_thing
    }
  }
}

[Davier]

How about having a way to add marker components when loading? Something like this:

let handle = server.load_with("my_texture.png", MyTexture).unwrap();

Then it can easily be filtered when an event AssetEvent is received.

By the way, I think these "event systems" would be great for the UI systems too.

[TheRawMeatball]

Re event systems: I'd rather not have these, as I think they share a lot of the same flaws as for-each systems. I'd rather not see a repeat of that.

[cart]

Yeah I'm also not super sold on event systems. They don't really seem worth it to me.

[kabergstrom]

Hmm I can think of a few paths forward here:

1. Listen for AssetEvent::Created, then read the dependent textures from the loaded asset and update them accordingly

2. Listen for each AssetEvent::Created, then somehow filter it down. The AssetServer stores asset dependency metadata, but that isn't currently exposed and it doesn't store "reverse dependencies",  so you would need to do a full scan to find parents and walk "up" the tree.

3. Give Loaders the chance to override the settings of dependencies when they are loaded. Atelier Assets (soon to be renamed Distill) supports dependency configuration via asset.meta files. I'm not sure if theres a way for loaders to override their dependent asset configuration, but we could consider adding that feature. (im curious what @kabergstrom thinks of this)

(1) seems like the easiest option to me and should work right now in Bevy. (3) seems like a valuable feature to have, but couldn't happen immediately.

Modifying a dependency's data would be incorrect in the face of multiple assets depending on the same modified asset. A correct approach would be to generate a new asset based on an import-time dependency, then modify the data of the new asset. Distill's Importer cannot access data from other Importers directly yet, but I am working on adding a way to read asset artifacts and arbitrary key-value pairs from Importers of other source files. This will be the way to implement automatic generation of atlases for example.

As for the issue raised in the original post, the solution in Distill is to modify a source file's Importer::Options struct, serialized in the .meta file, which enables the Importer to change how the asset is created as it is read from the source file.

[alice-i-cecile]

Relevant to this, see #1272 for a possibly more efficient / clearer way to handle systems that should run when there's events to be processed.

[inodentry]

EDIT: the original tone of this post was somewhat hostile. I have edited it to make it more constructive and on-topic.

@cart event systems are merely a slightly-nicer syntax for what we already have and do not provide the desired benefits.

Most of the boilerplate is still required, such as handling the event, and looking up the asset using the handle.

For example, let's say I want to do some processing for a specific texture. Here is what it looks like with an event system:

// need to store the handle somewhere to be able to find my texture
struct MapTexture(Handle<Texture>);

fn process_map_texture(
    In(event): In<AssetEvent<Texture>>,
    map_handle: Res<MapTexture>,
    mut assets: ResMut<Assets<Texture>>,
    // ... maybe I need more system params?
) {
    if let AssetEvent::Created { handle } = ev {
        if *handle == map_handle.0 {
            let texture = assets.get_mut(handle).unwrap();
            // i can finally do something with `texture`
        }
    }
}

fn setup_textures(server: Res<AssetServer>, commands: &mut Commands) {
    let handle = server.load("map.png").unwrap();
    commands.insert_resource(MapTexture(handle));
}

---

I feel it shouldn't be necessary to deal with events or asset storage at all, to do this. Or require a resource (or another mechanism) for tracking the handle.

It seems like a very common use case to want to do some processing on specific assets, when they get loaded (the event is overkill, i don't care about modifications or removals, and i don't care about other assets).

Now, here is what it could look like with the syntax proposed in the OP:

fn process_map_texture(
    // get the asset directly, and the handle just in case we need it
    In((texture, handle)): In<(Texture, Handle<Texture>)>,
    // ... maybe I need more system params?
) {
    // i can just do something with `texture`
}

fn setup_textures(server: Res<AssetServer>) {
    let handle = server.load("map.png").unwrap();
    server.set_processor(handle, process_map_texture.system());
}

---

AssetEvents are certainly useful, but they are for systems that care about detecting changes to assets in general. I am sure they are awesome for bevy's internal systems that upload new data to the GPU when textures get modified, and other similar use cases. This post is not about replacing asset events with something else.

But for this asset-processing use case (and this is a common use case), they are just cumbersome to use. I strongly believe we need something specialized, to make it ergonomic. I think it would fit into the spirit of bevy, which is to provide "refreshingly simple" and low-boilerplate APIs.

[cart]

I know what you're asking for ... rest assured that I read and assimilated your post. You want an asset-system-specific lower boilerplate way to modify assets when they load.

I wasn't dismissing your proposal or even suggesting that we shouldn't do it. I was merely (1) pointing out that the ergonomics of the current approach have improved recently (2) responding with an alternative that is slightly more ergonomic than what we have now and comparatively smaller in implementation complexity and "api surface".

Please understand that I am now in a situation where people are submitting requests for large changes to the way we write engine code on a regular basis. I believe some amount of pushback on "domain specific ergo improvements at the cost of increased api surface" is healthy for the project. You might think that this specific feature is clearly correct and worth the tradeoffs, but you also aren't on the hook for maintaining it over the long haul, whereas I am.

Also understand that while I have a thick skin and don't personally care about the condescending tone you chose to adopt in the post above, I would not tolerate that when deployed on someone else.

[cart]

On the topic of "event systems" I agree that they are a marginal improvement (which also happens to come at the cost of api surface). I'd largely like to have a conversation about things in that "vein". If the "asset load interceptor pattern" can't be solved with ideal ergonomics using the generic tools we have today, are there any other generic tools we can build to make that easier?

Maybe we won't find a good answer to that question. But lets talk about it?

[inodentry]

Also understand that while I have a thick skin and don't personally care about the condescending tone you chose to adopt in the post above, I would not tolerate that when deployed on someone else.

OK, sorry for that. I did not expect it to come across that way. No disrespect or offense was intended. I just felt the need to exaggerate, because I felt misunderstood (your previous post felt dismissive). I'll be more mindful of reacting like this in the future. Anyway...

you also aren't on the hook for maintaining it over the long haul, whereas I am.

I absolutely respect your position.

... people are submitting requests for large changes to the way we write engine code on a regular basis. I believe some amount of pushback ... is healthy for the project

I totally agree. I am aware that it is tempting for many of us to want to reimagine how bevy works, because we are so in love with it, and I understand how that isn't healthy.

I believe any drastic or absurd proposals should best be used as a source of inspiration to draw from, and we should moderate them down to something reasonable and actionable. Try to distill the essence of what they are going for, to see if the benefits can be achieved with incremental changes to existing things.

This is how I personally try to approach these discussions. I think we are on the same page about this.

If the "asset load interceptor pattern" can't be solved with ideal ergonomics using the generic tools we have today, are there any other generic tools we can build to make that easier?
... lets talk about it?

Yes, please! I am not in any way insisting that my proposed solution is the best one. If we can come up with something that fits better into bevy, or is more generally-useful, while, critically, providing the same desired practical benefits, that would be amazing!

My proposed solution is just the only one I could think of so far, that actually provides the desired benefits.

I wasn't dismissing your proposal or even suggesting that we shouldn't do it. I was merely (1) pointing out that the ergonomics of the current approach have improved recently (2) responding with an alternative that is slightly more ergonomic than what we have now and comparatively smaller in implementation complexity and "api surface".

On the topic of "event systems" I agree that they are a marginal improvement.

Anyway, I hope you understand why I felt that your post was dismissive. It simply pointed out a slight syntax improvement that does not actually address the issue.

Again, sorry for the misunderstanding. I hope we cleared it up.

[cart]

Awesome sounds like we're on the same page / we both misinterpreted each others intents to an extent. In retrospect I might have over-policed on tone in this case / there was nuance here that was near a line but not quite on it. But I like to be direct when it comes to my expectations for discourse in Bevy communities. If I can misinterpret intent, other people (who are more vulnerable than me) can too. The less room for error we have, the better (which it sounds like we totally agree on). Thanks for being understanding / responding in a professional way.

Its past my bedtime so I'm signing off for now 😄

Feel free to keep the conversation going / share any ideas you have

[inodentry]

Well, I think you are right about the tone. It was unnecessarily emotionally charged in a way that wasn't beneficial to the intention. I am happier now that I have edited it.

Glad to know we are on the same page.

I think the influx of grandiose change proposals is something inevitable, as the community grows, because bevy is something that makes a lot of people very passionate and inspired. Even when bevy is stabilized, there will be people posting proposals asking to totally redesign bevy or make some drastic change to the core, because they think they know better. I understand your frustration, believe me. 😄 I don't want to contribute to it. If anything, I want to help.

When I see such posts, I try to figure out what are the underlying benefits that the author actually wants, and if I can provide a response that, ideally, suggests how similar benefits might be achievable with less drastic changes. Or if not, at least highlight/clarify those underlying benefits, so that the discussion can focus on them and the author feels validated. It's a tricky thing; by trying to "push back" on proposals, you could come across as dismissive or hostile to the author.

Anyway, this is off-topic for this discussion.

[inodentry]

It seems to me that there are several categories of things one might want to do with an asset:

1. Specify various parameters (such as the texture sampling parameters, as in the OP).
2. Modify the data in a substantial way (such as applying some modification to a texture's pixels)
3. Autogeneration of more assets (such as building a texture atlas out of textures)
   ... anything else?

(1) is by far the most common use case. Sounds like, in Distill, such parameters are stored in metadata files, which are easy to edit. That sounds very nice and eliminates the need to poke them from code. Seems like an all-round better solution.

(2) and (3) are both specialized enough that there wouldn't be a large number of such systems in a project, so they probably wouldn't mind the extra boilerplate.

So, seems like an all-powerful code-injection mechanism like I previously proposed might not actually be that useful, to justify the implementation effort and complexity.

However, as we aren't (yet?) using Distill with its metadata files, a more ergonomic way to do (1) in existing bevy would be nice. Maybe we should focus the discussion on that specifically?

(also, (1) was the use case that originally motivated this discussion, but when writing the OP, i was just thinking "why not make it as powerful and versatile as possible if we can", so I expanded the scope)

Are there any alternative ways in Bevy for doing things like setting texture sampler parameters (and other asset metadata? what other asset types have metadata?), without having to write a system to manipulate the asset after it is loaded?

[cart]

@alec-deason is working on the integration now: https://discord.com/channels/691052431525675048/749332104487108618/803339744851263561

[cart]

Or rather, has been working on it and might pick it up again soon

[alec-deason]

I actually expect to start picking at it again this week.

[inodentry]

Yes, considering that Distill is designed to be a fully-featured asset management solution with the goal to provide sane and productive ways to do all the reasonable things you'd want to do with assets, it seems like the way to go.

I don't remember if it was certain / decided that integrating Distill is the plan for bevy. @cart ?

If that's the case, and the integration with Distill isn't too far out into the future, then I guess we don't really need to do anything now (and we can close this discussion).

Any API changes around assets better wait until we have Distill and we get some experience with what that workflow feels like.

[cart]

Yup it is still the plan. You can track progress here: #708

[CooCooCaCha]

Has there been any discussion around a jobs/tasks system? I'm imagining an API where you can dynamically submit jobs at runtime that get scheduled along with regular system execution. For this specific use-case you could tell bevy to load an asset and then dispatch a job to process the asset after it has loaded.

[alice-i-cecile]

We could hook into the existing bevy_tasks subcrate for this :)

[TheRawMeatball]

Hmm, I don't think hooking into bevy_tasks is a very good idea since its a dependency of bevy_ecs, and that means if we implement this stuff using bevy_tasks, we won't have any access to world / resources etc.