wasi preview 2 support #4027

dgryski · 2023-12-04T21:07:29Z

This PR adds -target=wasip2 support to TinyGo. It no longer depends on wasi-libc, but. has its own miniature libc that turns the calls into appropriate component calls.

src/runtime/runtime_wasm_wasi.go

aykevl · 2023-12-05T12:20:15Z

Interesting!
Suggestion: use -target=wasip2 to more clearly specify the WASI version.

src/runtime/runtime_tinygowasmp2.go

dgryski · 2023-12-08T03:16:49Z

Now we just need to unstub the 29 calls in libc_wasip2.go with Go implementations of the associated C calls using preview2 calls under the hood.

src/syscall/env_wasip2.go

ydnar · 2024-02-22T20:42:07Z

@dgryski the HOWTO in this PR might need to be updated to reflect the new build steps (and no need for a Preview 1 adapter!):

tinygo build -target=wasip2 -x -o main.wasm ./cmd/wasip2-test
wasm-tools component embed -w wasi:cli/command $(tinygo env TINYGOROOT)/lib/wasi-cli/wit/ main.wasm -o embedded.wasm
wasm-tools component new embedded.wasm -o component.wasm
wasmtime run --wasm component-model component.wasm

dgryski · 2024-04-02T21:37:58Z

This is now in a state where it's roughly equivalent to wasip1 support. I still plan to add networking support, but that can come in a later PR.

src/syscall/libc_wasip2.go

ydnar

Phone review, WIP

ydnar · 2024-04-03T16:35:36Z

loader/goroot.go

@@ -240,6 +240,7 @@ func pathsToOverride(goMinor int, needsSyscallPackage bool) map[string]bool {
 "internal/fuzz/": false,
 "internal/reflectlite/": false,
 "internal/task/": false,
+ "internal/wasm/": false,


Do we still need this?

loader/goroot.go

main.go

dgryski · 2024-04-24T05:46:48Z

@deadprogram Yup, I'm looking at the linux.yml workflow now..

.github/workflows/linux.yml

tools/tgtestjson.sh

deadprogram

It is very large, but it is also the result of a lot of iterating and is covered by tests.

I say we merge and then continue from there.

@aykevl any comments?

aykevl

Read through most of the PR, and skimmed libc_wasip2.go. Looks reasonable to me though I have a few concerns here and there.

And some more comments that I can't really do inline:

Can you add wasip2 to builder/builder_test.go? That way we can be sure the LLVM flags etc are the same between Clang and TinyGo.
I see a lot of places where you use //go:export. This is equivalent to //export, and deviates to from upstream Go. I'd prefer using //export everywhere.
(I should never have added //go:export in the first place, I was just being lazy...)

GNUmakefile

builder/build.go

compiler/symbol.go

aykevl · 2024-05-03T12:00:34Z

compiler/testdata/errors.go

-// ERROR: //go:wasmimport modulename invalidparam: unsupported parameter type int
-// ERROR: //go:wasmimport modulename invalidparam: unsupported parameter type string
-// ERROR: //go:wasmimport modulename invalidparam: unsupported parameter type []byte
-// ERROR: //go:wasmimport modulename invalidparam: unsupported parameter type *int32
-//
-//go:wasmimport modulename invalidparam
-func invalidparam(a int, b string, c []byte, d *int32)


Why remove these tests? I think they're important.

They're mostly no longer valid with the relaxed types in wasm args. We can re-add the slice test.

I'd prefer if these tests were updated instead of removed altogether. ABIs are tricky to get right, and I'd like to have some tests for them.
For example: slices, maps, structs, and func values are still not allowed.

To make it really perfect, it'd be nice to add some tests to compiler/testdata/pragma.go to show that the compiler indeed produces the expected LLVM IR for the given types. Not a blocker though.

src/runtime/runtime_wasm_wasip2.go

src/syscall/env_libc.go

src/syscall/errno_wasip2.go

src/syscall/syscall_libc_wasi.go

src/syscall/libc_wasip2.go

Fixes feedback in tinygo-org#4027.

Fixes feedback on tinygo-org#4027.

Addresses PR feedback in tinygo-org#4027.

all: address PR feedback

dgryski · 2024-05-07T16:26:25Z

@aykevl PTAL. I think we've addressed everything.

deadprogram · 2024-05-07T17:45:59Z

@dgryski see https://github.com/tinygo-org/tinygo/pull/4027/files#r1589100941

fix: remove extra wasm-tools file copies

deadprogram · 2024-05-07T19:13:34Z

@aykevl any other comments before we merge?

aykevl

Having a proposal eases my worries about relaxing the //go:wasmimport rules, though I still have a few small things about the code that I'd like to see improved. (I'm careful with this because experience tells me that once the compiler allows something, it's very difficult to restrict it again in the future - it's much easier the other way round if needed!).

Other than that, it's mostly just minor nits.

aykevl · 2024-05-09T19:01:23Z

compiler/symbol.go

+ case *types.Struct:
+ return true


This is actually not part of golang/go#66984 anymore (though it is when it's a pointer-to-struct and pointer-to-array, this probably needs some more logic):

Supporting struct and [...]T by value

A previous version of this proposal included support for passing struct and [...]T types by value by expanding each field recursively into call parameters. This was removed in favor of a simpler initial implementation but could be readded if users require it.

Also, now I think about it, we should definitely be stricter here and also check every field in the struct. Otherwise func(x *struct{field [2]int}) would be legal which it shouldn't be.

See: golang/go#66984 (comment)

aykevl · 2024-05-09T19:04:05Z

compiler/symbol.go

+ case types.String:
+ return true


Note that this works more or less by accident (because the way both Go and TinyGo represent strings is the most obvious way to do it so they match).

This also matches how WASI 0.2 represents strings (ptr, len pair)

compiler/symbol.go

aykevl · 2024-05-09T19:08:12Z

compiler/testdata/errors.go

-// ERROR: //go:wasmimport modulename invalidparam: unsupported parameter type int
-// ERROR: //go:wasmimport modulename invalidparam: unsupported parameter type string
-// ERROR: //go:wasmimport modulename invalidparam: unsupported parameter type []byte
-// ERROR: //go:wasmimport modulename invalidparam: unsupported parameter type *int32
-//
-//go:wasmimport modulename invalidparam
-func invalidparam(a int, b string, c []byte, d *int32)


I'd prefer if these tests were updated instead of removed altogether. ABIs are tricky to get right, and I'd like to have some tests for them.
For example: slices, maps, structs, and func values are still not allowed.

To make it really perfect, it'd be nice to add some tests to compiler/testdata/pragma.go to show that the compiler indeed produces the expected LLVM IR for the given types. Not a blocker though.

aykevl · 2024-05-09T19:10:24Z

compiler/testdata/errors.go

@@ -16,14 +16,6 @@ type Uint uint32
 //go:wasmimport modulename validparam
 func validparam(a int32, b uint64, c float64, d unsafe.Pointer, e Uint)


Might be a good idea to update this to add some new types (like string and *struct{}).

src/runtime/runtime_wasm_wasip2.go

aykevl · 2024-05-09T19:13:56Z

src/runtime/runtime_wasm_wasip2.go

+// libc constructors
+//
+//export __wasm_call_ctors
+func __wasm_call_ctors()


I'd prefer if it was removed then (though that's not a blocker, it just removes noise).

src/runtime/runtime_wasm_wasip2.go

dgryski · 2024-05-16T21:11:20Z

Is there anything else left unaddressed before we merge?

aykevl · 2024-05-17T14:59:46Z

@dgryski yes there is, namely with //go:wasmimport. We are currently incompatible with the upstream proposal for loosening //go:wasmimport:

The upstream proposal doesn't allow passing structs as-is.
Also, the proposal now requires structs.HostLayout markers in every struct to ensure they match the underlying platform. The only thing TinyGo needs to do here is to check that the field is present: I haven't checked specifically but I'm pretty sure we already match the Canonical ABI for the in-memory layout.
We don't validate whether struct fields are allowed. So for example the following code would be allowed, even though it clearly is invalid:
```
//go:wasmimport foo bar
func bar(x *struct{a func()})
```

We could decide to deviate from the upstream proposal (especially when it comes to passing structs by value) but if we do that I'd like to see at least a good explanation for that (and the comment should be more than just "this reflects the relaxed type restrictions proposed here" because it doesn't).

ydnar reviewed Dec 4, 2023

View reviewed changes

src/runtime/runtime_wasm_wasi.go Outdated Show resolved Hide resolved

johanbrandhorst reviewed Dec 5, 2023

View reviewed changes

src/runtime/runtime_tinygowasmp2.go Outdated Show resolved Hide resolved

dgryski force-pushed the dgryski/wasi-preview-2 branch 2 times, most recently from 399c173 to eeb8f09 Compare December 8, 2023 02:42

jordan-rash reviewed Dec 8, 2023

View reviewed changes

src/syscall/env_wasip2.go Outdated Show resolved Hide resolved

dgryski force-pushed the dgryski/wasi-preview-2 branch 2 times, most recently from 4288422 to 681281a Compare December 16, 2023 06:42

dgryski force-pushed the dgryski/wasi-preview-2 branch from 1cb14ca to 90028f7 Compare January 30, 2024 20:30

dgryski force-pushed the dgryski/wasi-preview-2 branch from 2279be1 to 6b0b0ef Compare February 22, 2024 18:59

dgryski force-pushed the dgryski/wasi-preview-2 branch 2 times, most recently from 630f6e5 to dcea7bb Compare February 28, 2024 22:51

dgryski force-pushed the dgryski/wasi-preview-2 branch 9 times, most recently from 567da72 to b2ae42f Compare April 2, 2024 21:34

dgryski marked this pull request as ready for review April 2, 2024 21:37

lpereira reviewed Apr 2, 2024

View reviewed changes

src/syscall/libc_wasip2.go Outdated Show resolved Hide resolved

lpereira reviewed Apr 2, 2024

View reviewed changes

src/syscall/libc_wasip2.go Outdated Show resolved Hide resolved

lpereira reviewed Apr 2, 2024

View reviewed changes

src/syscall/libc_wasip2.go Outdated Show resolved Hide resolved

ydnar reviewed Apr 3, 2024

View reviewed changes

dgryski force-pushed the dgryski/wasi-preview-2 branch from 532f31d to 72e73c9 Compare April 4, 2024 19:00

dgryski force-pushed the dgryski/wasi-preview-2 branch from cd32f3b to 589e1ec Compare April 24, 2024 18:07

deadprogram mentioned this pull request May 1, 2024

os/user.Lookup #4217

Open

dgryski force-pushed the dgryski/wasi-preview-2 branch from 3e61972 to 19780b4 Compare May 1, 2024 19:09

deadprogram reviewed May 2, 2024

View reviewed changes

.github/workflows/linux.yml Show resolved Hide resolved

deadprogram reviewed May 2, 2024

View reviewed changes

.github/workflows/linux.yml Outdated Show resolved Hide resolved

all: wasip2 support

8f9b569

dgryski force-pushed the dgryski/wasi-preview-2 branch from 19780b4 to 8f9b569 Compare May 2, 2024 13:50

deadprogram reviewed May 2, 2024

View reviewed changes

tools/tgtestjson.sh Show resolved Hide resolved

deadprogram approved these changes May 2, 2024

View reviewed changes

aykevl reviewed May 3, 2024

View reviewed changes

ydnar and others added 10 commits May 4, 2024 12:18

syscall: remove !wasip2 build tag

cb87062

Fixes feedback in tinygo-org#4027.

syscall: enforce NULL termination of C string returned from getcwd

34eea86

Fixes feedback on tinygo-org#4027.

compiler: add link to golang/go#66984

035960a

Addresses PR feedback in tinygo-org#4027.

Merge pull request #14 from ydnar/ydnar/wasip2

8ab1a61

all: address PR feedback

src/syscall: go:export -> export

55a7543

src/syscall: document what libcErrno refers to

e6cb9a9

GNUmakefile: remove unused make targets

304c62e

src/runtime: remove useless error check

9196300

builder: add waisp2 to builder tests

454a85d

main: make wit flags available for tinygo run

c5154c7

fix: remove extra wasm-tools file copies

1a4962a

Merge pull request #15 from omnide/fix-wasip2-no-copy

7a1b925

fix: remove extra wasm-tools file copies

aykevl reviewed May 9, 2024

View reviewed changes

src/runtime: remove __wasm_call_ctors() for wasip2

3ef711a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

wasi preview 2 support #4027

wasi preview 2 support #4027

dgryski commented Dec 4, 2023 •

edited

aykevl commented Dec 5, 2023

dgryski commented Dec 8, 2023

ydnar commented Feb 22, 2024

dgryski commented Apr 2, 2024

ydnar left a comment

ydnar Apr 3, 2024

dgryski commented Apr 24, 2024

deadprogram left a comment

aykevl left a comment

aykevl May 3, 2024

ydnar May 3, 2024 •

edited

aykevl May 9, 2024 •

edited

dgryski commented May 7, 2024

deadprogram commented May 7, 2024

deadprogram commented May 7, 2024 •

edited

aykevl left a comment

aykevl May 9, 2024

aykevl May 13, 2024 •

edited

aykevl May 9, 2024

ydnar May 10, 2024

aykevl May 9, 2024 •

edited

aykevl May 9, 2024

aykevl May 9, 2024

dgryski commented May 16, 2024

aykevl commented May 17, 2024

		@@ -16,14 +16,6 @@ type Uint uint32
		//go:wasmimport modulename validparam
		func validparam(a int32, b uint64, c float64, d unsafe.Pointer, e Uint)

wasi preview 2 support #4027

Are you sure you want to change the base?

wasi preview 2 support #4027

Conversation

dgryski commented Dec 4, 2023 • edited

aykevl commented Dec 5, 2023

dgryski commented Dec 8, 2023

ydnar commented Feb 22, 2024

dgryski commented Apr 2, 2024

ydnar left a comment

Choose a reason for hiding this comment

ydnar Apr 3, 2024

Choose a reason for hiding this comment

dgryski commented Apr 24, 2024

deadprogram left a comment

Choose a reason for hiding this comment

aykevl left a comment

Choose a reason for hiding this comment

aykevl May 3, 2024

Choose a reason for hiding this comment

ydnar May 3, 2024 • edited

Choose a reason for hiding this comment

aykevl May 9, 2024 • edited

Choose a reason for hiding this comment

dgryski commented May 7, 2024

deadprogram commented May 7, 2024

deadprogram commented May 7, 2024 • edited

aykevl left a comment

Choose a reason for hiding this comment

aykevl May 9, 2024

Choose a reason for hiding this comment

Supporting struct and [...]T by value

aykevl May 13, 2024 • edited

Choose a reason for hiding this comment

aykevl May 9, 2024

Choose a reason for hiding this comment

ydnar May 10, 2024

Choose a reason for hiding this comment

aykevl May 9, 2024 • edited

Choose a reason for hiding this comment

aykevl May 9, 2024

Choose a reason for hiding this comment

aykevl May 9, 2024

Choose a reason for hiding this comment

dgryski commented May 16, 2024

aykevl commented May 17, 2024

dgryski commented Dec 4, 2023 •

edited

ydnar May 3, 2024 •

edited

aykevl May 9, 2024 •

edited

deadprogram commented May 7, 2024 •

edited

Supporting `struct` and `[...]T` by value

aykevl May 13, 2024 •

edited

aykevl May 9, 2024 •

edited