Unparsing with tree-sitter?

[nalzok]

I wonder if tree-sitter supports "unparsing", in the same sense as ast.unparse in Python. Basically, "parsing" converts a chunk of source code to an AST, whereas "unparsing" converts an AST back to a chunk of source code.

If not, what's the best way to generate such an "unparser" from grammar files? A high-level overview would suffice. Thanks!

[ahelwer]

Not supported. You can use python (or any other language) to read the grammar.json file in the src directory to get the meaning of each AST node, and generate code from that. Would require some extra manual per-language work if an external scanner is involved.

[ahlinc]

As @ahelwer said it isn't supported in the same sense of like it exists in Python because Tree-sitter doesn't store original text pieces in all leaf nodes, instead of that it just stores ranges that every node occupy in an original text.
It depends what is a purpose of unparsing:

• If to restore a piece of source text based on some node than it's easy to get a needed node position as a byte range and get a slice of input related to the node range.
• If to render to a text some modified AST, then it's not supported because Tree-sitter was build around idea of immutable trees.

[Laegel]

@nalzok You're looking for code generation given an AST. Under the hood, Python's unparse generates code by consuming the AST and adding Python-specific syntax elements: it's tedious and sounds very time consuming.
Universal code generation based on grammar definitions is something I've been trying to implement with Tree Sitter but there are many limitations and it requires, as stated by @ahelwer, more work to be done than just reading the files.
I've been stuck with this idea because you can't just pick a grammar and generate a reliable AST definitions. I say "reliable" because while it's totally possible, the AST will be unpractical with a poor API and missing parts.
For example, take this simple Python code:

async def foo(bar):
    return bar

which results in the following s-expression tree:

As you can see, async keyword doesn't appear in the AST: it belongs to the parse tree and isn't considered relevant in the AST because it's a syntax token. However, on the developer level, this keyword has a meaning. If we were to implement an AST API to create functions, we would definitely need a way to specify if it's async or not.
It's very likely that you would use this AST API as an intermediate layer between Tree Sitter and the code, which means you would also need to turn the AST/CST node into your AST API node.
I don't think Tree Sitter (or any other parser generator like ANTLR) was made for this purpose. There are other issues that I won't list here but IMO it's probably needed to (at least!) rewrite the grammars to have a data-driven approach code generation.
I won't meddle in this discussion any further, if anybody's interested in this topic, please contact me. :)

[ahlinc]

Currently the tree-sitter-python grammar doesn't fully cover modern Python syntax and needs to be improved and properly define all terminals to make them visible as anonymous or named tokens.

[verhovsky]

Python switched to a PEG parser

https://medium.com/@gvanrossum_83706/peg-parsing-series-de5d41b2ed60

https://devguide.python.org/internals/parser/

My understanding is that this is a more powerful type of parser than what tree-sitter is, so it's possible that some parts of the Python grammar can never be fully, correctly supported (without reworking tree-sitter).

[ahlinc]

Python's grammar is not a problem for Tree-sitter, all its hard cases can be managed with an external scanner.
I know only two real problems:

1. Processing macro defs and application points in languages like C.
2. Languages that allow postponed things declarations where a type (how to interpret) can't be derived from the syntax.

[conartist6]

There is a parser out there made for this purpose, but it's currently in alpha under the codename cst-tokens. My goal was/is to take the best ideas from tree-sitter, ANTLR, and SrcML and mash them up into one immensely powerful tool. That tool captures and classifies every single token (even whitespace) to ensure that the immutable CST it produces can be used by an editor as state, and also has a pleasant serialization format (wip) which ensures that it is trivially possible to do streaming reads, writes, and transforms of serialized trees.

[conartist6]

I took a ton of the same ideas, like syntax-driven tokenization. My grammars aren't pure data structures though, they are made up of executable code which yields transitions to a VM which evolves LR state. All in all my algorithm is most analogous to Earley parsing but because I have the full power and expressiveness of code at parse-time I never need to fall back to an external tokenizer, even for gnarly syntaxes like heredoc.

[berchn]

#1657 implements an "unparser" on top of tree-sitter for original and modified ASTs. The implementation is non-trivial--the grammar is embedded into the classes to reproduce the source, and class definitions and parse trees are programmatically modified to store additional information based on the grammars.