Bevy for embedded platforms (retro consoles)

[IamTheCarl]

I enjoy home brewing for older consoles. I enjoy embedded in general.
An ECS for such a machine would be very interesting, as I can see a few interesting advantages to the architecture on an older console, and I don't think there has ever been an opportunity to use ECS on such a console since it's a relatively new concept.

I was hoping to talk a bit about the idea of running the core of Bevy on embedded systems, including retro consoles from around the mid90s to 2010 range. I don't think running on such devices should be a high priority goal for Bevy at the moment because many of the features needed are not yet stable Rust. I also think the goal should not apply to all crates in the project, as most would be irrelevant or impractical in such an environment. I think just getting the ECS alone to work in such an environment would be revolutionary for retro homebrew.

The console I am most familiar with is the Game Boy Advanced. I'm going to use that console as my primary example, but the constraints of the hardware I'm going to mention are pretty universal to all embedded platforms.

Memory

Organizing the usage of memory maps

Many systems of this type have different kinds of memory. In the GBA you have three types of memory:

1. Internal working memory: This is the fastest memory in the system. The only constraint is that there isn't much of it, so you should use it for data that is accessed most frequently, perhaps several times per frame. The only down side is that there isn't much of it (in the case of the GBA, 32kb)
2. External working memory: It's slower than the internal working memory, but you have a lot more of it. Use this for data that is only accessed maybe once per frame. Its main advantage is that there's a lot more of it (256kb)
3. ROM: This is the slowest memory, having only a 16 bit data bus, meaning each read operation takes twice as long. All data here can be accessed with a simple &static _ so I don't think Bevy really needs to worry about this. Data that needs frequent/fast access (including executable code) is often copied into one of the working memory areas. I don't think any of these things should be managed by Bevy.

This is a very common kind of setup for embedded systems. Some will include sections of memory with verification mechanisms, for safety critical data. This RAM is typically useful to game projects and will likely just get wrangled in with the rest of the system memory.

Typically a GBA project will have two allocators, one for the IWM, and another for the EWM. Enabling developers to pick which allocator to use would be a powerful boon, and likely the biggest advantage to using an ECS on a console like this. I think the ability to select storage with #[component(storage = "SparseSet")] could revolutionize the usage of these two allocators, making it easy to store frequently accessed components in the "InternalTable" and infrequently accessed components in the "ExternalTable". All these tables could just be the standard one but backed by different allocators.

Avoiding Out of Memory conditions

With memory spaces being 32kb or 256kb in size, it's easy to run yourself out of memory if you get careless. The fact that you can have "partially loaded" entities by simply removing components is a huge advantage.

Portability

You can run any game on a PC with the appropriate emulator but it would be nice to have native ports. With an ECS being so modular, it would be easy to swap out your GBA graphics and sound systems with PC graphics and sound systems. You would need to swap out your "InternalTable" and "ExternalTable" allocators to both just use the default allocator. For a game designed to run on 288kb of RAM and maybe a 32Mb ROM cartridge, it should be easy for such a game to run on even old PCs.

What it would take to make this possible

A quick browse through the source code, a major constraint that I can see at the moment is that I can't add custom storage without modifying this struct and the way it is referenced.

Rather than speculate on what other issues there could be, I decided to just try building bevy_ecs for the Game Boy Advance.
I cloned the agb template and added bevy_ecs = { version = "0.12.0", default-features = false } to the list of dependencies.

I encountered the following problems:

• armv7 does not support atomic pointers, so the AtomicPtr type is not available. Because of that, types under alloc::sync and alloc::task are unavailable. That's a bit of an odd one since as far as I understand a single core CPU should only need synchronization to protect data structs from interrupts (this is typically implemented by disabling interrupts whenever a mutex is locked). All pointer operations can be considered atomic since an interrupt can't interrupt mid-instruction execution. They'll either wait until the instruction completes or cancel the instruction and jump back to it later, depending on the platform and its configuration.
• Some dependencies access std directly, rather than going through core. This is by far the most common error I got.

These two issues were the bulk of it, but there was over 5000 errors and I was not willing to read each one individually.

I think the idea of bevy_ecs in an embedded environment is an interesting idea, possible, and benificial, but I don't think it should be a high priority for the team, if a priority at all. I started this discussion so I could assess why someone would want this functionality and what would be needed to make it a reality.

Related discussions

It seems that there was a push to make Bevy no_std friendly in the past but was cancelled: #705
I don't think we should make all of Bevy no_std so I agree with the discussion, but just the ECS being no_std would be powerful for the reasons explained above.

If the Bevy team disagrees then that's fine. This can be archived and used as documentation of why it's difficult to make the project no_std.

[alice-i-cecile]

I'm happy to support any PRs that swap over from std to core, or similar trivial changes :) For more complex changes, doing them one at a time using a std config flag for regressions would be helpful.

I think alloc is mandatory though, although I don't think that's a problem for your platforms.

[IamTheCarl]

I agree. Removing the dependency on alloc is unreasonable, especially when it's so easy to just add an allocator in Rust.

Most of the work is probably going to be in the dependencies of bevy_ecs. The ones related to threading seem to be a heavy point of pain. Being able to swap out the "executor" or "scheduler" (I'm not sure what you call it) for a single-threaded one is probably the easy solution here since a lot of these targets aren't capable of running hardware threads.

[IamTheCarl]

Okay, so I did more digging tonight.

bevy_ecs has 13 direct dependencies.

It seems that tracing's dependence on Arc, which depends on AtomicPtr, is killing the idea of a no_std bevy_ecs.
These kinds of systems disable interrupts as an alternative to atomic pointers. If we could implement that ourselves it could be done, but looking around on the unofficial Rust Discord, it looks like that's currently not possible.

With that major issue plus some others I found, my motivation for this is kinda killed, but I'll post my notes for all the libraries I did figure out for those who may want to know:

bevy_ptr

No modifications necessary.

bevy_reflect

bevy_math

No modifications necessary.

bevy_reflect_derive

No modifications necessary.

bevy_utils

bevy_utils

bevy_ptr

bevy_ptr

bevy_tasks

TODO

bevy_utils

I always feel uneasy when I see a "utils" thing, because that's usually just a box for random things people felt were too small to deserve their own space. I'm unsure what bevy_ecs needs this for, but it looks to be a significant hurdle.

It seems bevy_ecs uses bevy_utils for tracing, SyncCell, and a few other things.

ahash

getrandom

Not looking good. Depends on syscalls to get random numbers from the OS. I'm not sure what bevy_ecs would need that for and am hoping we can just hide it behind a feature flag.

tracing

Disable default features and you'll get most of the way there. This has already been done but for some reason we explicitly enabled std. I'm not yet sure why.
The killer for this one though is that it depends on atomic pointers, which not all platforms support (including the GBA).

instant

Heavy dependency on SystemTime, which has dependence on the OS. I don't think we can make this one no_std. We'll have to see if we can feature gate it.

uuid

TODO

hashbrown

TODO

bevy_utils_proc_macros

TODO

petgraph

TODO

thiserror

this_error

nonmax

TODO

async-channel

TODO

event-listener

TODO

thread_local

TODO

fixedbitset

TODO

rustc-hash

TODO

downcast-rs

TODO

serde

Disable default features, enable alloc and derive features.

thiserror

Replace with "thiserror-core", a drop-in replacement fork of thiserror that uses a Rust nightly feature to support no_std. Then disable default features.

[rlidwka]

It seems that tracing's dependence on Arc, which depends on AtomicPtr, is killing the idea of a no_std bevy_ecs.
These kinds of systems disable interrupts as an alternative to atomic pointers. If we could implement that ourselves it could be done, but looking around on the unofficial Rust Discord, it looks like that's currently not possible.

Can you clarify how these systems do concurrency, and what alternatives exist instead of atomic pointers? Or is it all single-threaded, so they don't need it?

[IamTheCarl]

I'd be happy to answer that.
I'm going to assume you don't know much about embedded so sorry if this is over-detailed.

Most embedded systems do not have concurrency. The ones that do (which are not retro consoles) provide atomic operations, so this problem doesn't exist there.

In a non-concurrent embedded system, the only synchronization you need to worry about is interrupts. Interrupts are special functions. When a certain hardware event (such as a timer completing or a button being pressed) happens, the code that is currently being executed will be paused, a return pointer pushed on the stack, and the CPU will jump to the interrupt handler function. When that function completes, the return pointer is popped off the stack and the original function is resumed. Do note that on some platforms, interrupts can interrupt interrupts, so you need to make sure synchronization is properly implemented in those as well.

Synchronization issues exist for data that is shared between both interrupts and main code. Wrapping them in a critical section is the usual solution here. A critical section is a block of code where interrupts are disabled, thus meaning that an interrupt won't be able to manipulate data until the critical section exits.

If we implemented our own atomic pointers they'd just have all their operations wrapped in critical sections.
The show stopper here is that as far as I can tell, there is no way for us to implement AtomicPtr ourselves.

[IamTheCarl]

Putting more thought into this, portable atomic type libraries already exist. These will use the standard atomics on tier 1 platforms and use fallbacks on lesser platforms. If we replaced Arc with something that's backed by these, we could have a maintainable codebase that can run on those platforms.

The pain in the butt is that we'd have to get upstream packages we depend on to use these portable atomic types.

[bjorn3]

getrandom

Not looking good. Depends on syscalls to get random numbers from the OS. I'm not sure what bevy_ecs would need that for and am hoping we can just hide it behind a feature flag.

Getrandom has a feature to specify your own random number source: https://docs.rs/getrandom/latest/getrandom/macro.register_custom_getrandom.html

[seurimas]

Just figured I'd add to this with my own efforts to get bevy running in no_std: https://github.com/seurimas/bevy

Many changes in that fork may be unnecessary, particularly the ones affecting now-optional crates like bevy_reflect/bevy_tasks.

I'm targeting the Playdate (https://github.com/seurimas/bevy_playdate), building off of the crank/crankstart libraries to get all that running. Getting Bevy to work in no_std was mostly an effort in removal and falling back on alloc and third-party drop-ins for not(feature = "std") cases. I've only just "finished" the hello_world example, so I don't know how well these drop-ins will actually work in real use.

Some take aways:

bevy_reflect, bevy_a11y, bevy_log (easy fixes)

I had to make all of these optional features in various crates. It's possible bevy_reflect could have been fitted out for no_std, but the many, many std-default crates involved made the prospect seem like more effort than it was worth.

I think bevy_reflect could be more properly optional in a lot of crates in the real Bevy, even for non-no_std purposes. The changes are fairly minimal, and some people might want to avoid the overhead. Additionally, some crates already make a point of making bevy_reflect optional.

bevy_tasks (easy fix?)

I've not really put my application through its paces well enough to say for sure that removing this is safe. Time will tell, but it seems that tasks are mostly used by other crates that are not included.

Petgraph (upstream, but possibly on their radar already)

petgraph is currently not properly no_std compatible, due to dependencies having default features, mostly. As such, I had to make those changes, which could be merged back in, as they seem open to supporting no_std: https://github.com/seurimas/petgraph

web_time/instant (upstream, no good solution unless we supply our own)

We need Instant::now, but that's not supported outside of std. I had to implement my own: https://github.com/seurimas/instant

Obviously, this isn't a broadly applicable solution. Until we get some sort of register_instant_now macro to define our own Instant::now when we need it, this is probably the hardest stopper against full no_std support in real Bevy.

macros, toml_edit (upstream, no good solution)

toml_edit seems firmly against supporting no_std, which means some macro helper functions are missing, particularly the translations from, for example, bevy_ecs to bevy::ecs. This is hacked around in my crate by just tacking use bevy::ecs as bevy_ecs at the top, and hardcoding the resolution in the utils to use the supplied crate name, rather than trying to translate it to a different syn::Path.

The macros were just blamed for std violations in other crates, I think. As per discussion below, the macros can have std just fine.

Please excuse the mess.

I'm new to no_std applications, having only started looking into it when I got my Playdate for Christmas. As such, a lot of my changes are a bit messy. I believe bevy could cleanly and with minimal overhead begin to support no_std with only a subset of the changes I have in my branch. Some of the std -> alloc and std -> drop-in swaps could be placed somewhere central, like bevy_utils does for hashbrown::HashMap.

Good luck to anyone else looking to get Bevy running on embedded platforms! I hope my work might make that a little bit easier for you.

[seurimas]

It does seem that the Playdate has that support, yes. At least, there were no problems with it at compile time.

[seurimas]

You may be able to drop-in replace portable_atomic for your own purposes, though.

[IamTheCarl]

I did consider that. It could solve most issues but it doesn't have a replacement for Arc.
Besides, I'm not entirely sure this should be considered Bevy's problem. It's more a limitation in the Rust language itself. Being able to custom-implement atomic types on these kinds of platforms would be very helpful.

[seurimas]

You could maybe define your own build target, with your own atomics implementation, but thats a whole different topic.

[bjorn3]

Note that rust has defined atomics to be lock-free. As such you can't emulate rust atomics without hardware support unless you are on a single core target and do tricks like disabling all interrupts during the critical section or undo/finalize an atomic operation inside the interrupt handler whenever an interrupt happens. If you are on a multi core target or not running as part of the OS itself to do interrupt tricks, you are out of luck.

[mrchantey]

I'm super excited for this, bevy_ecs is an awesome architecture for embedded platforms.

I've just been able to fit it onto an esp32-c3 with wifi and a simple motor / depth sensor program in an std environment using esp-idf-sys etc. Its tight but it works! 🚀

[Maximetinu]

@seurimas was you finally able to run bevy_ecs on the Playdate? (I mean, in your forked Bevy. I'm interested in this as well)

[Maximetinu]

Thx!

I've given it a try, pulling down crankstart, your bevy fork and the bevy_playdate crate, and adding bevy_playdate as a dependency. I noticed that your forked the instant crate as well, so I did that. Same for the petgraph crate. However, I get this error on the latter error: failed to select a version for petgraph the package bevy_utils depends on petgraph, with features: "graphs" but petgraph does not have these features

I cannot see any commit from you on your petgraph fork, and there is no "no_std" feature, so..., maybe you fix it as well but didn't pushed it?

[seurimas]

Aha, you're right! I've pushed my instant and petgraph forks.

[Maximetinu]

Thanks! I pulled them down, I have everything now, but I still get this error when trying to run the bevy_playdate hello world example

   Compiling bevy_playdate v0.1.0 (/path/to/bevy_playdate)
error[E0433]: failed to resolve: use of undeclared crate or module `bevy_ecs`
 --> src/sprites.rs:7:10
  |
7 | #[derive(Component, Deref, DerefMut)]
  |          ^^^^^^^^^^^^^ use of undeclared crate or module `bevy_ecs`
  |
  = note: this error originates in the derive macro `bevy::ecs::component::Component` (in Nightly builds, run with -Z macro-backtrace for more info)
help: consider importing this module
  |
1 + use bevy::ecs::component;
  |

I'm trying to debug why, use bevy::prelude::*; is certainly at the top of the src/sprites.rs file, and adding use bevy::ecs::component or even fully qualifying the component derive like bevy::ecs::component::Component doesn't work

[seurimas]

Did you pull my bevy branch again? I checked in a small change there, alongside the other forks, that might be relevant. There was some churn with how macros were handled, as alluded to above. You could also try something like use bevy::ecs as bevy_ecs. It's been a while, but that was the sort of fix I had to use at some point.

[Maximetinu]

I was missing that last commit, now it works!

Thanks a lot!