Go bindings for language grammars

[dcreager]

The smacker/go-tree-sitter repo is a popular set of Go bindings around the core tree-sitter library. It also includes a selection of languages grammars directly in the repository, making it very ergonomic to use tree-sitter to parse those particular languages from Go. In particular, the way that @smacker is vendoring the generated C files for each language grammar means that everything plays very nicely with go mod and go build, which is perfectly willing and able to compile C code as part of the build, but doesn't want to delegate to external tools like tree-sitter generate or make. The per-language Go wrapper code is nice and small, and boilerplate.

The individual language grammars already include bindings for Node and Rust, which can be auto-generated as part of tree-sitter generate. That makes it easy for us to publish NPM and Rust packages for individual languages, e.g. tree-sitter-python [npm] [rust].

The individual language grammars currently have their generated C source code checked into git. It should be easy to add the small amount of Go wrapper code as a new binding that is automatically created by tree-sitter generate. That would mean that go-tree-sitter could be simplified to only contain the Go API around tree-sitter itself, and users of that library could use go mod to depend on particular language grammars (and particular pinned versions of those grammars, if needed), instead of delegating that decision to go-tree-sitter.

If you only consider what I typed above, I think this proposal is really compelling. But unfortunately it depends on checking the generated C code into the language repos. We've discussed not doing this moving forward: #730 (comment).

So now we have conflicting requirements: there are real usability wins to not checking the generated code into git, but on the other hand, that would preclude us from (easily, automatically) creating Go modules for each of the language grammars.

Thoughts? Opinions? Other concerns I haven't thought of?

[ahelwer]

This treads dangerously close to designing a new package format, but what about emitting the generated tree-sitter code in a zipped release? It wouldn't necessarily even be tightly bound to github. Everyone has access to a CI system these days, which could run on checkin and generate/upload the code. tree-sitter generate could even create CI files for commonly-used CI systems.

Anyway I support the idea of adding go bindings during tree-sitter generate until it's decided what to do with the generated files. My repo is already over a gigabyte in size!

[tclem]

There's definitely an n x m problem here of how to manage language parsers (tree-sitter-<LANG> repositories) and bindings (<BINDING_LANG>-tree-sitter repositories). From the perspective of a consuming application, I like the idea of using that application's dependency system to pull in tree-sitter and the desired tree-sitter parsers.

smacker/go-tree-sitter is really nice, but one challenge is that if you want to write a custom parser or use another parser in the ecosystem you have to get that added to smaker/go-tree-sitter instead of being able to just take on a new dependency in your application. I think having a canonical list of well supported parsers for a binding language is great, but there will always be cases where applications want to pull in a parser that the rest of the community might not want or care about. Another use case is just testing newer versions of a parser or delaying upgrading to the latest version until some bug gets fixed.

In terms of committing generated C code, I think that warrants a separate discussion.

[dcreager]

In terms of committing generated C code, I think that warrants a separate discussion.

I worry that they're not separate discussions, though — if we add Go bindings to the language repos, which depends on having the generated code checked into git, that limits our ability to stop checking the generated code into git. We'd either have to (a) figure out how to store the generated code in some other place that works with go mod, or (b) break the per-language Go bindings moving forward.

At the very least, we need to be aware of this, so that it's a mindful decision if we do decide to add the per-language Go bindings.

[maxbrunsfeld]

Does Go not have any way of installing packages besides cloning git repositories? I think you’re right that it doesn’t, but I haven’t followed the language closely lately.

one solution would be to only include generated code in certain tagged commits, and not on the development branch.

[tclem]

Go modules can be served by a module proxy which must return a .zip with some specific constraints. Consuming go modules from version control systems (e.g. git) is likely more canonical for open source projects, but go doesn't require that.

[smacker]

smacker/go-tree-sitter is really nice, but one challenge is that if you want to write a custom parser or use another parser in the ecosystem you have to get that added to smaker/go-tree-sitter instead of being able to just take on a new dependency in your application.

Thank you for the kind words.

I want to clarify that the lib was designed in a way that you can use separate packages for language grammars.
Example of such a grammar https://github.com/shagabutdinov/go-tree-sitter-ruby (before it was added to the main repo).
Or you can just bundle necessary grammars to your project: smacker/go-tree-sitter#49

The repo contains common grammars for convenience and simplicity but I'm completely open to move them out.