Use tree-sitter in Rust, compile everything to WASM?

[jrieken]

Assume I have a library that is written in Rust and that uses tree-sitter via its rust bindings. Next, I want to compile that library to WASM and expose its features via wasm_bindgen.

As a sample I have the snippet below and I try to compile it via cargo b --target wasm32-unknown-unknown (without success). Did anyone try this? Should this be asked in some Rust forum or somewhere else?

use tree_sitter::Parser;
use wasm_bindgen::prelude::*;

#[wasm_bindgen]
pub fn foo() -> u64 {
    Parser::new().timeout_micros()
}

Published a simple sample here: https://github.com/jrieken/rust-wasm (run cargo b --target wasm32-unknown-unknown, easiest with dev containers)

/cc @dcreager

[liamrosenfeld]

The ABI incompatibility between Clang and Rust for wasm-unknown-unknown causes issues for this. The easiest way around that currently is to use the Rust port of tree sitter here: https://github.com/shadaj/tree-sitter-c2rust/tree/c2rust-master

[kamoshi]

Is there a similar thing available for Treesitter Highlight? It would be useful to be able to run Treesitter highlighting in a wasm setting

[liamrosenfeld]

I think that project also includes a translated tree sitter highlight. I haven't tested it though since what I'm working on involves highlighting already parsed trees which is not a feature of normal tree sitter highlight. So I'm using a modified version that just lives in a file in my project.

[mainrs]

That only works for the base tree-sitter package. As soon as one wants to include a language, like Julia, the compilation fails again.

Can a similar approach be used for languages I want to use inside of WASM?

[kamoshi]

I've tried to compile tree-sitter, tree-sitter-highlight, and some languages, via rustc to wasm-unknown-unknown, however the problem is that the source code for tree-sitter contains #include <stdio.h>. This header is not available while compiling for wasm-unknown-unknown, and it can only be compiled with emscripten, which is a quite problematic target in rustc

[liamrosenfeld]

I got around that issue for the grammars by providing what was missing through shim files that are added when compiling dependencies though environment variables (wasm-sysroot/ and justfile here: https://github.com/cacticouncil/lilypad).

However, if you're using tree-sitter from rust for wasm32-unknown-unknown, you're gonna run into the ABI incompatibility between rust and C. You can join the club of waiting on this to merge to finally fix it: rust-lang/rust#117919

[benwis]

@liamrosenfeld I was hoping you could tell me a bit more about what you meant in the comment above. I too want to compile treesiter into a wasm file that can be loaded on the web. I'm confused about what you meant by using tree-sitter from rust for wasm32-unknown-unknown. Sorry for bugging you!

[liamrosenfeld]

If you just want to use wasm compiled tree sitter in the browser from javascript, you can follow this. What I've said here is about compiling a Rust project that depends on tree sitter to web assembly (there is currently an issue with the Rust compiler that requires some workarounds when doing so).

[mainrs]

This has been merged some days ago. Is there another issue that I can track or get involved in to add direct compilation to Rust?

[liamrosenfeld]

Yeah now that rust-lang/rust#117919 has merged, you can build it on nightly. However, there is a small import issue in tree sitter that my PR here fixes #3304. When using that fix, the only thing that is needed is a shim C standard library (like an extended version of https://github.com/cacticouncil/lilypad/tree/main/wasm-sysroot)

[mainrs]

Yeah now that rust-lang/rust#117919 has merged, you can build it on nightly. However, there is a small import issue in tree sitter that my PR here fixes #3304. When using that fix, the only thing that is needed is a shim C standard library (like an extended version of cacticouncil/lilypad@main/wasm-sysroot)

Would this also work for tree-sitter-lang packages? I tried patching them with a custom patch.crates-io entry, but that didn't work. Maybe the shim was missing. It couldn't find stdlib.h.

[liamrosenfeld]

Currently, to provide the shim to the cargo dependencies, I'm setting CFLAGS_wasm32_unknown_unknown := echo "-I$(pwd)/wasm-sysroot" at build time.

[benwis]

It's a bit different, but I was able to compile treesitter and its parsers as part of my markdown to html compiler crate femark to wasm via the wasi target, and deploy in the browser. I wrote about that process here: https://benw.is/posts/compiling-rust-to-wasi

…

On Mon, Apr 22, 2024, at 8:03 AM, Sven wrote: > Yeah now that rust-lang/rust#117919 <rust-lang/rust#117919> has merged, you can build it on nightly. However, there is a small import issue in tree sitter that my PR here fixes #3304 <#3304>. When using that fix, the only thing that is needed is a shim C standard library (like an extended version of cacticouncil/lilypad@`main`/wasm-sysroot <https://github.com/cacticouncil/lilypad/tree/main/wasm-sysroot>) > Would this also work for `tree-sitter-lang` packages? I tried patching them with a custom `patch.crates-io` entry, but that didn't work. Maybe the shim was missing. It couldn't find `stdlib.h`. — Reply to this email directly, view it on GitHub <#1550 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVBTCNX7GXA5T7R3BNJ6CLY6URFXAVCNFSM5KQKWJUKU5DIOJSWCZC7NNSXTOKENFZWG5LTONUW63SDN5WW2ZLOOQ5TSMJZGAZDCNY>. You are receiving this because you commented.Message ID: ***@***.***>

[StratusFearMe21]

I had luck with using this version of tree-sitter, and putting a tree-sitter generated parser in the path of the c2rust transpiler, and linking it directly to the tree-sitter library. However, for the parser I'm compiling, it takes a comically long time to transpile (around 10 minutes on my machine), and also to compile the generated rust code. The parser code I put in came out as an 8MB rust file, so the long compile time definitely scans.

[StratusFearMe21]

Yeah. Ill post the code here later today

[StratusFearMe21]

@mainrs Here is the source code and instructions for getting tree-sitter parsers working under Rust WASM. Enjoy 😄 !

https://github.com/StratusFearMe21/tree-sitter

[StratusFearMe21]

@mainrs Fixed an error in the README (if you've already gotten started with the repo)

[mainrs]

Thanks! This only works as long as I want to include a single language, right? If I want to use it with Java and Haskell, then this method would no work.

[StratusFearMe21]

The only thing stopping you from including more languages is your computer's RAM. c2rust will run out of memory if you ask it to transpile too many parsers

Say we want to include another tree-sitter parser, and we put it in lib/src/another_parser.c. These are all the changes you would need to make to the repo to include that new parser. From there, the rest of the instructions in the repo will apply like normal

diff --git a/lib/binding_rust/core_wrapper/mod.rs b/lib/binding_rust/core_wrapper/mod.rs
index c31e9899..9c0a34fd 100644
--- a/lib/binding_rust/core_wrapper/mod.rs
+++ b/lib/binding_rust/core_wrapper/mod.rs
@@ -17,6 +17,8 @@ pub mod core {
 
     pub use c2rust_bitfields::*;
 
+    #[allow(unused, clippy::all)]
+    pub(crate) mod another_parser;
     #[allow(unused, clippy::all)]
     pub(crate) mod api_raw;
     #[allow(unused, clippy::all)]
@@ -105,6 +107,7 @@ pub mod core {
 
     pub use alloc::*;
 
+    pub use another_parser::*;
     pub use api_raw::*;
     pub use get_changed_ranges::*;
     pub use imported_parser::*;
diff --git a/lib/src/lib.c b/lib/src/lib.c
index f9d15740..0fda74e6 100644
--- a/lib/src/lib.c
+++ b/lib/src/lib.c
@@ -1,6 +1,7 @@
 #define _POSIX_C_SOURCE 200112L
 
 #include "./alloc.c"
+#include "./another_parser.c"
 #include "./get_changed_ranges.c"
 #include "./imported_parser.c"
 #include "./language.c"
diff --git a/meson.build b/meson.build
index 68352513..18120da8 100644
--- a/meson.build
+++ b/meson.build
@@ -16,6 +16,7 @@ tree_sitter_lib = library(
   'lib/src/processed/tree_cursor.c',
   'lib/src/processed/wasm_store.c',
   'lib/src/processed/imported_parser.c',
+  'lib/src/processed/another_parser.c',
   include_directories: ['lib/include', 'lib/src', 'lib/src/wasm']
 )
 
diff --git a/process_c2rust/src/main.rs b/process_c2rust/src/main.rs
index 6b51712c..48cb30f4 100644
--- a/process_c2rust/src/main.rs
+++ b/process_c2rust/src/main.rs
@@ -525,6 +525,7 @@ fn main() {
     );
     public_apis.insert("lib/src/wasm_store.rs", vec!["TSWasmStore"]);
     public_apis.insert("lib/src/imported_parser.rs", vec![]);
+    public_apis.insert("lib/src/another_parser.rs", vec![]);
 
     public_apis
         .keys()