Versioning and releases for tree-sitter grammar repos

[Will-Sommers]

👋👋👋 Heyo! Thanks so much for the project, really really liking it and digging into it.

This isn't in a tree-sitter-{lang} repo as it appears to be a common issue across the org. I've noticed that nearly all of the tree-sitter-{lang} repos are currently on versions that were mostly cut in March of 2021. Making a wildly successfully project probably comes with a lot of dev and maintenance.

• Are there approaches for consistent tag releases that y'all are considering?
• Can we help out at all?

Thanks again!

[maxbrunsfeld]

Hi Will! Yes, the project has experienced some growing pains as the amount of open source contribution has increased. I'd really love to improve the situation, but I have limited time to work on these issues right now. Some possible interested parties:

• @dcreager, @hendrikvanantwerpen, @rewinfrey et al at GitHub
• @Luni-4 and @calixteman at Mozilla
• @aryx and @mjambon at R2C

In particular, @Luni-4 has suggested to me some ways of using GitHub actions to improve the situation (see tree-sitter/tree-sitter-java#98). Hi Luni-4! I'm really sorry for not taking any action on your suggestion yet; I've just been very busy these last several months.

I'm wondering if you folks using Tree-sitter at bigger companies can help set some direction on how we want to do releases and versioning and such. It seems like there are some other folks using Tree-sitter who are willing to help out if we can decide on steps to take.

[dcreager]

Thanks for opening this discussion, @Will-Sommers, it's been a topic that's been front of mind recently.

Are approaches for consistent tag releases that y'all are considering?

For the individual language grammars, we've found that tagged version numbers have actually been less helpful than we assumed they would be. We're relying on pinned dependencies on specific commit SHAs for each language grammar repo that we're consuming, such as:

[dependencies]
tree-sitter-python = { git = "https://github.com/tree-sitter/tree-sitter-python", rev = "0d17ed665458c49eb9c6e95155e196a85bff67b7" }

Or maybe said a different way, it could still be useful to tag versions, but I don't think we should choose version numbers that try to align with the releases of tree-sitter itself. If we go back to tagging version numbers for tree-sitter-python, for instance, those version numbers should describe meaningful changes to the Python grammar itself, and not try to describe which tree-sitter versions the pregenerated grammar is compatible with. (Because of the ABI compatibility that @maxbrunsfeld put into the underlying C code, each grammar is compatible with a rather large range of tree-sitter versions; see e.g. tree-sitter/tree-sitter-python#129.)

What we found — at least right now while there's still a bit of churn in the grammars — was that basically every merged PR was changing a grammar, or its highlight or tagging rules, enough that it would deserve a semver major version bump. So for now we've just been using pinned SHAs.

[Will-Sommers]

Heyo @dcreager and @maxbrunsfeld.

Thank you for the responses. I opened this discussion primarily to understand what a desirable future looks like in your minds.

From Douglas' response, it sounds as if y'all will ultimately choose to release a major version at some point in the future, but not now. Until then, we should rely on SHAs and treat upgrades akin to major versions or sub v1.0.0 patches(which they are), with much caution.

For what it is worth, the project I'm beginning to contribute to is using SHAs and things are going fine.

Other notes:

Or maybe said a different way, it could still be useful to tag versions, but I don't think we should choose version numbers that try to align with the releases of tree-sitter itself.

Agreed.

What we found — at least right now while there's still a bit of churn in the grammars — was that basically every merged PR was changing a grammar, or its highlight or tagging rules, enough that it would deserve a semver major version bump

Would any update to a grammar, which returns a differently named or shaped parse tree, be a breaking change in your mind?

[dcreager]

Would any update to a grammar, which returns a differently named or shaped parse tree, be a breaking change in your mind?

That's a great question ­— the core question, in fact! I think all of the other decisions in this thread will be mostly obvious once we figure out this part.

Broadly speaking, I believe there are two kinds of consumers of tree-sitter grammars: "direct" and "indirect".

Indirect consumers don't depend on the contents of the grammar. A syntax highlighting program, for instance, might have a direct dependency on the tree-sitter-tags crate. It also needs to load in several grammars, each of which should include highlighting rules, but it only cares that each language's grammar and highlighting rules are consistent with each other. It doesn't need visibility into the particular grammar node names or syntax tree shapes coming out of the parser. The highlighting rules do care about that, but they should hopefully be updated in sync with grammar changes as part of a single PR.

An indirect consumer doesn't care about version numbers at all. Depending on how strongly reproducible they want their builds to be, they'll either want to pin to a specific SHA, or use the latest and greatest regardless of version number (>=0 in Cargo, for instance).

Direct consumers, on the other hand, do care about the specific node names and tree shapes, because those are referenced somewhere by name — maybe directly in code, maybe in a declarative file (like highlighting rules). But if anything in the grammar changes, those consumers would be out of date and no longer work — which tells me that should be a breaking semver change. Or, on the assumption that every change needs careful review, even if it turns out not to be breaking, the direct consumer should use pinned SHAs.

[pokey]

That's a nice way of thinking about it @dcreager. @Will-Sommers I think I'd argue that Cursorless is a "direct consumer", as we rely on specific patterns in the parse tree, so we should prob just continue pinning SHAs for now

Btw @dcreager I saw your talk at UCSC's LSD Seminar. Such cool stuff!

[dburriss]

Yes @dcreager the SemVer is really helpful when you are consuming multiple tree-sitter languages. I actually came to point out that the use case between just using tree-sitter, or a single language is quite different to using lots of languages in a single project as we do in rust-code-analysis. I like the (in-)direct language. Suppose I am just reiterating that projects will prefer different things not just based on the direct grammar dependency but also based on 1 or many tree-sitter deps.

[Luni-4]

Hello everyone and thank you for opening this discussion @Will-Sommers! @maxbrunsfeld yep, I supposed you were busy, but no problem at all, let's try to find a solution that could be suitable for each of us!

Well, the main problem we are experiencing in rust-code-analysis with SHAs is related to the cargo publish command. You can build and distribute binaries through GitHub using the SHA approach, but you can't publish your own crate on crates.io, since it is required to set a specific semantic version for each of your dependencies.
That's why, to reduce the burden for a developer, we have thought a GitHub Action that automatically publishes a grammar on crates.io. So when a maintainer decides to bump a new grammar version, as soon as a new tag is inserted, the action starts and executes the cargo publish command verifying before that everything is correct with the --dry-run option.

Why we think this approach could be helpful:

• You can publish your own crate on crates.io, impossible to do with the SHA approach
• Using crates directly from crates.io helps a lot maintaining third-party projects, since you don't have to look for specific SHAs around to make your project work
• The private key necessary to upload on crates.io can be hidden behind a GitHub Secret, so anyone except the owner of a repository can see that
• Allows to publish a grammar on crates.io with a reduced effort from a developer (just a git tag number command) and the maintainability for the action is very low. The only requested effort is inserting the private just after adding the action
• Avoid unwanted forks, created with the sole purpose of publishing their own version of the grammar on crates.io in order to publish their own crates. One fork less means no duplicated crate and more space on crates.io
• The approach can be applied to each grammar indifferently

As developer, I'm willing to add the action for each grammar, unfortunately I can set a secret environment variable because I don't have privileges.

Feedback and critiques about this approach are appreciated! Thanks!

[hendrikvanantwerpen]

Let me try to draw a conclusion from the previous threads (one and two):

1. Having version numbers on grammars is necessary for some use cases, such as publishing to crates.io, where SHA's are not a viable alternative.
2. Grammar versions should be independent of Tree-sitter versions, and follow this semantic versioning scheme:
   • Changes that change the structure or node names of the parse tree require a major version bump.
   • Other changes can be minor version bumps. This can include changes to highlighting and tagging rules, increasing the range of supported Tree-sitter versions.
   • Direct consumers of parse trees depend on specific major versions, indirect consumers (e.g., that use highlighting or tagging output) may accept any version if they wish.
3. Reusable automation like an Action can be used to make publishing new grammar versions to crates.io or NPM as simple as pushing a new version tag.

Regardless of the consensus we reach, I think it would be useful to have a section on the Tree-sitter website documenting these practices for grammars, as a reference for maintainers and contributors.

[Luni-4]

Can we set up a deadline for this one? For example, one month from now?

[kylo252]

I've been working for a while on integrating tree-sitter into vcpkg:

• [tree-sitter] add new port microsoft/vcpkg#26058
• [tree-sitter] add multiple ports microsoft/vcpkg#26272

This is mainly to solve the issue of distributing parsers, by utilizing CMake's capabilities for cross-platform support (based on #1822)

Take a look https://github.com/kylo252/tree-sitter-registry for a demo of how simpler and more robust the workflow can be with the help of vcpkg.

Highlights

• cross-platform support powered by cmake and vcpkg
• allows both dynamic and static builds
• strict ABI versioning, but can still be overridden with --overlay-ports=
• helper function for integrating additional parsers, see ports/tree-sitter-common

Extra goodies from using vcpkg

• much, much faster builds using assets caching
• packaging and distribution support using binary caching
• no vendor lock-in, the core functionality is a collection of CMake scripts
• integration with cargo is available through https://crates.io/crates/vcpkg

[dcreager]

This has come up again in a couple of places, e.g.:

• chore(master): release 1.0.0 tree-sitter-regex#19
• Consider releasing 0.21 (and yanking 0.20.2 and 0.20.3) due to a breaking change tree-sitter-javascript#294

Since we're rehashing this discussion down in grammar repos, I thought it would be good to escalate here again to see if we can come to a consensus.

Broadly speaking, there seem to be two self-consistent strategies for publishing and consuming grammars, and as tree-sitter maintainers we should choose which one to endorse/enforce:

1. All bets are off: By far the majority of grammar changes affect the parse trees that are produced, and so consumers should pin to specific commits in the grammar repo. It is an active decision to bump to a new grammar version, which is likely to require manual intervention. This is especially true if they are using queries (or anything else that depends on the tree structure) that are not developed in the grammar repo itself.

2. Semver koolaid tastes great: Not every grammar change affects the parse tree; it would be useful to convey information about which ones do; and the world has standardized on semver as the best way to do that.

From what I can tell, it seems that consensus among the maintainers is to prefer (1). @maxbrunsfeld @ahlinc @amaanq, do you agree? Any other maintainers I should mention to weigh in?

[For the record, I prefer (2), but I more prefer to have a decision made and recorded somewhere, so if more maintainers prefer (1) I'll defer to that.]

---

Now, on the assumption that we decide to go with (1), I also want to bring up a corollary for discussion. tree-sitter/tree-sitter-javascript#294 came up as a specific issue because the version numbers we are publishing to crates.io do not line up with the "all bets are off" guarantees. When people see a patch release to a x.y.z version number, they are likely to assume that they can upgrade without having to change anything. And more importantly, an automated tool like cargo is also going to assume that, unless you are careful to use = requirements for all of your grammars.

So that means we either have to carefully document everywhere that we are not making semver guarantees with our version numbers (and somehow get all of our downstream consumers to notice and pay attention to that). Or we should choose version numbers that do line up with our guarantees — for instance, a date-based version like YYYYMMDD. (That's a valid semver where every future release is considered incompatible, so cargo will never try to upgrade without your approval, even if you don't remember to use an = requirement.)

[ahlinc]

@dcreager please don't tag me related to Tree-sitter. I don't want to be related to such worse releases like v0.20.9, because half of required packages wasn't version bumped at all but the git tag was issued, that was just a CLI release without other parts. There are no one with whom to build consensus and I decided to left the project: #2797 (comment)

[clason]

As nvim-treesitter maintainer (arguably the biggest consumer of tree-sitter grammars on the web with over 250 supported languages and counting), I am strongly in favor of 2 and have been pretty obnoxious about that in the past.

In a nutshell, semver signals compatibility to downstream, which here is queries. So it's straightforward (if your repo contains queries, which there should be no reason not to) to verify whether a change

1. breaks parsing current queries (-> major version bump)
2. requires query changes to make use of new features (-> minor version bump)
3. doesn't affect queries (-> patch version bump).

(Yes, that means you'll quickly end up with Chrome-level version numbers. There are worse things.)

I'll add that this will be of limited use, though, as long as tree-sitter does not support parser introspection -- in particular, checking its version -- which would allow me to detect an incompatible query before YOLO parsing it and having to sweep up a heap of angry red errors.

When people see a patch release to a x.y.z version number, they are likely to assume that they can upgrade without having to change anything. And more importantly, an automated tool like cargo is also going to assume that, unless you are careful to use = requirements for all of your grammars.

I'll also add that semver has an explicit proviso for major version zero, which covers this behavior. It's cargo that is wrong here. (And in fact tree-sitter serves a much wider ecosystem than just Rust.)

That doesn't mean that datever is not a reasonable versioning strategy, mind you; but I do think we can do better and convey meaningful semantic information.

[dcreager]

I'll also add that semver has an explicit proviso for major version zero, which covers this behavior. It's cargo that is wrong here. (And in fact tree-sitter serves a much wider ecosystem than just Rust.)

That doesn't mean that datever is not a reasonable versioning strategy, mind you; but I do think we can do better and convey meaningful semantic information.

This is why I cast it as a corollary — the first decision is what guarantees we want to make / information we want to provide. Only then can we decide how best to convey those guarantees to (all of) our tooling. I don't think it's relevant that cargo is "wrong", though I really do hope that we can avoid needing different grammar version numbers for cargo, npm, etc due to different inconsistent readings of the semver spec! 😰

[clason]

I don't know how you can "read differently" https://semver.org/#spec-item-4...

(But agreed; first we need to agree on -- introspectible! -- parser versions as an official thing. Right now we do not look at parser version at all since it's basically useless; we follow the "lock-and-pray" strategy you mentioned together with some tooling that allows us to pull in non-breaking updates in a semi-automated way. Of course, this means we can miss other kinds of regressions or useful parser improvements.)

[maxbrunsfeld]

It sounds like most downstream consumers prefer that we try to use semver in a more fine-grained way in grammar repos. I'm fine with adopting that policy.

What code and documentation changes do we need to make in order to begin implementing this policy?

[clason]

parsers associate to languages (via filename or symbol); neovim handles the mapping to filetype. but that part is irrelevant to what we're talking here.

The take-home message is that Neovim (must) ingest compiled parsers anywhere on a list of "runtimepaths". So no API, no dice.

Same with queries; so if we can inspect a parser's version, we can just skip loading incompatible queries, no matter where they come from. (Including colorschemes. Yes, really.)

[maxbrunsfeld]

Ok, if I understand correctly, it sounds like at some point, there was a design decision in Neovim to have Tree-sitter parser .so files be loaded as self-describing runtime files, using only their filename to identify them. And that design would be difficult to change at this point, because that plugin interface is already very widely used.

That's definitely a valid reason to add the feature. (Reminds me of all the crap we went through to evolve the Atom plugin API). I just want to be clear on what problem we're solving.

[clason]

(Not really much of a choice, given the general structure of things.)

[texastoland]

CC @alexdima re. microsoft/vscode#147648 (comment) microsoft/vscode#50140

[Thaodan]

I was wondering something similar. When packaging treesitter compiled grammars with rpm. All the metadata doesn't really transfer into the api.
For example some grammars have two variants which each have their own metadata but after compilation there is no information inside the compiled grammar file besides the version.