Allow env/args/preopens to exceed 64k in size #8594
Merged
+452
−336
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit fixes an issue with the wasip1 adapter published with Wasmtime which current limits the size of environment variables, arguments, and preopens to not exceed 64k. This bug comes from the fact that we more-or-less forgot to account for this when designing the adapter initially. The adapter allocates a single WebAssembly page for itself but does not have a means of allocating much more than that. It's technically possible to continue to call
memory.grow
or possiblycabi_realloc
from the original main module but it's pretty awkward.The solution in this commit is to take an alternative approach to how these properties are all processed. Previously arguments/env vars/preopens were all allocated once within the adapter and stored statically. This means that after startup they're copied from where they reside in-memory, but the downside is that we have to have enough memory to hold everything. This commit instead tries to "stream" the items so they're never held entirely within the adapter itself.
The general idea in this commit is to use the "align" parameter to
cabi_import_realloc
to figure out what's being allocated and route the allocation to the destination. For example an allocation with alignment 1 is a string and can go directly into a user-supplied pointer where an allocation with alignment 4 is a pointer-based allocation which must be stored within the adapter, but only temporarily.With this redesign it's now possible to have the sum total of args/envs/preopens to exceed 64k. The new limitation is that the max-length string plus size of the max length of these arrays must be less than 64k. This should be a more reasonable limit than before where any one individual argument/env var is unlikely to exceed 64k (or get close).
Closes #8556