$schema usage stats in summer 2023

[SorinGFS]

After analyzing all the schemas on schemastore.org and the referenced or incorporated subschemas, I got the following results:

Number of evaluated files: 2301
Size of evaluated files: 93Mb

json-schema.org's own files were not included.

Detailed result:

$schema	count	percent
http://json-schema.org/draft-04/schema#	560	30.20%
http://json-schema.org/draft-04/schema	233	12.57%
http://json-schema.org/draft-06/schema#	33	1.78%
http://json-schema.org/draft-06/schema	2	0.11%
http://json-schema.org/draft-07/schema#	799	43.10%
http://json-schema.org/draft-07/schema	116	6.26%
http://json-schema.org/draft/2019-09/schema#	2	0.11%
http://json-schema.org/draft/2020-12/schema	1	0.05%
http://json-schema.org/schema#	5	0.27%
https://docs.renovatebot.com/renovate-schema.json	1	0.05%
https://json-schema.org/draft-04/schema#	6	0.32%
https://json-schema.org/draft-07/schema#	9	0.49%
https://json-schema.org/draft-07/schema	23	1.24%
https://json-schema.org/draft/2019-09/schema#	2	0.11%
https://json-schema.org/draft/2019-09/schema	8	0.43%
https://json-schema.org/draft/2020-12/schema	36	1.94%
https://meta.json-schema.tools/	3	0.16%
https://raw.githubusercontent.com/DataDog/schema/main/.version/schema.json#	1	0.05%
https://raw.githubusercontent.com/aws-cloudformation/cloudformation-cli/master/src/rpdk/core/data/schema/provider.definition.schema.v1.json	1	0.05%
https://raw.githubusercontent.com/aws-cloudformation/cloudformation-resource-schema/blob/master/src/main/resources/schema/provider.definition.schema.v1.json	2	0.11%
https://raw.githubusercontent.com/aws-cloudformation/cloudformation-resource-schema/master/src/main/resources/schema/provider.definition.schema.v1.json	5	0.27%
https://raw.githubusercontent.com/k3d-io/k3d/main/pkg/config/config.versions.meta.schema.json#	1	0.05%
https://schema.cloudformation.us-east-1.amazonaws.com/provider.definition.schema.v1.json	5	0.27%
total	1854	100.00%

Usage grouped by target:

$schema	count	percent
draft-04	799	43.10%
draft-06	35	1.89%
draft-07	947	51.08%
2019-09	12	0.65%
2020-12	37	2.00%
total	1830	98.71%

I also analyzed the presence of dynamic refs in the files, the only files that contain them are:
-https://raw.githubusercontent.com/OAI/OpenAPI-Specification/main/schemas/v3.1/schema.json
-https://spec.openapis.org/oas/3.1/schema/2022-10-07

And those two files are in fact small variations of the same thing: OpenAPI v3.1.x documents without schema validation, as defined by https://spec.openapis.org/oas/v3.1.0

[Relequestual]

Thanks for doing this research.

It's worth noting here that using 2019-09 or 2020-12 is currently problematic, as most IDEs do not properly support it.

See just a few examples:

• Intellisense recommends invalid schema uris for JSON Schema drafts microsoft/vscode-json-languageservice#194
• Partial JSON Schema 2020-12 support results in failuer to highlight invalid schema features? microsoft/vscode-json-languageservice#149
• Autocompletion doesn't work with 2020-12 schema for sub levels microsoft/vscode-json-languageservice#179
• [json] JSONSchema draft 2019-09 support microsoft/vscode#98724

[SorinGFS]

(Read "editor" if you'd like, though VSCode is generally indeed considered an IDE for whatever the term means at this point.)

difference between Visual Studio Code and Visual Studio IDE?

I used to work in Visual Studio too, there when a project is started all their files are dynamically linked in a separate environment, and from that point on any change is automatically reflected everywhere. In vscode every change is basically a rescan of files.

[gregsdennis]

What was meant, Sorin, is that VSCode is a viable IDE these days for just about any language. It wasn't a comparison to Visual Studio.

[SorinGFS]

VSCode is a viable IDE these days for just about any language.

... but not for $dynamicRef! 😄 I didn't spend time to understand how vscode works under the hood, if someone did maybe should try to write some extension to solve this problem.

[gregsdennis]

No, it's an IDE. That means you can use it to develop and build programs.

You're correct in that it doesn't fully support JSON Schema, but JSON Schema isn't a programming language. Someone did write an extension: https://marketplace.visualstudio.com/items?itemName=prosser.json-schema-2020-validation

[SorinGFS]

It ...was a start, but obviously it didn't got too popular with 3k instals in 2 years. For people the point is to have something that they can rely on. Here is what someone says on the site you provided:

Lars Stockmann
5/11/2023
Nice idea, but seems unmaintained now. No changes for 2 years with open issues that prevent it from even loading, let alone working reliably in current VS versions.

Also, the author of that library wrote this warning:

[!WARNING] Schema substitution only works for $schema, not $ref. I may add this feature if there is enough demand, but that would require hooking deeper into the guts of the schema validation library I'm using.

Knowing how much time these things can consume I imagine the author gave up right on that point.

[benjagm]

most IDEs do not properly support it.

Shall we consider to contribute to those IDEs to get that support added?

[Relequestual]

I think it would take significant effort. I've offered multiple times to provide guidance and advice, but it's a matter of individual priorities for the people responsible. No one is telling them it needs to be fixed internally.

[jdesrosiers]

This is very interesting data. Thank you for sharing it. However, I'd like to add a little context so people don't draw the wrong conclusions.

It's important to keep in mind that data from schemastore.org is not representative of JSON Schema version usage as a whole. A very tiny fraction of schemas end up in schemastore.org. It only represents one small use-case for JSON Schema. The niche schemastore.org fills is that it's schemas are used by IDEs to provide hints for things like common configuration files (Example: package.json). This is just one use case for JSON Schema and a relatively minor one.

Other than representing only one small niche of JSON Schema usage, data from schemastore.org is skewed in a couple other ways. First, users are limited by what IDEs support. The unevaluated* keywords are extremely useful for this use case. If IDEs supported it, it's hard to believe that we wouldn't see vastly more use of 2019-09 and 2020-12 schemas in schemastore.org. The IDEs are forcing the use of less useful versions that people want to use.

The other way schemastore.org skews the results is that it's README tells people to use the oldest version they can when uploading schemas. This ends up being bad advice because many implementations are dropping support for older versions.

Given all that, the only conclusion I think can be drawn from this data is that draft-07 is the most recent version that's well and reliably supported in IDEs (which we already knew). I don't think this data can tell us anything about version trends in general and doesn't tell us what versions people prefer if they weren't limited by the IDEs.

Some additional observations from the data:

• I'm shocked to see how many invalid dialect URIs are used. There's no way those work reliably and indicate that they probably aren't actually ever used.
• I wonder how many of the draft-04 schemas are actually stale (never used or maintained). I imagine most are only draft-04 because they were written a very long time ago just haven't been updated because they either aren't used anymore or haven't needed to change. Data on when schema were added might be informative. If it turns out that people are still choosing draft-04 after draft-07 was supported by IDEs, that could tell us how much the README advise skews what version people choose.

[SorinGFS]

I'm shocked to see how many invalid dialect URIs are used. There's no way those work reliably and indicate that they probably aren't actually ever used.

There were also a fair number of 404 errors, I didn't count them. And, from the valid url's I also got 1422 invalid subschema url's. Still, this number was like less than 0.5% of total subschema links. The vast majority of subschema refs were valid.

I wonder how many of the draft-04 schemas are actually stale (never used or maintained). I imagine most are only draft-04 because they were written a very long time ago just haven't been updated because they either aren't used anymore or haven't needed to change.

You're right. Probably a schema once uploaded stays there forever, because I noticed apps having both, draft-04 and draft-07 schemas. They probably have the option to upload, but not to remove a schema once uploaded. I don't know if so, just speculate.

Data on when schema were added might be informative. If it turns out that people are still choosing draft-04 after draft-07 was supported by IDEs, that could tell us how much the README advise skews what version people choose.

I don't think their README can influence too much about the version choice. In general there are really elaborated schemas for mature apps, developed by large organisations. The compact version of the schema compiled using my implementation project is 122MB and has 1.45M rows. If anyone is interested to see what's in it please let me know and I will temporarily upload it somewhere. Also, if there are known API's containing schema links I'm interested to analyze them too.

[gregsdennis]

I don't think their README can influence too much about the version choice.

The readme definitely has an impact on what's uploaded, which affects what's available. That, in turn, does affect what JSON Schema versions are used and then what validators need to support.

It has become increasingly difficult to push JSON Schema forward when large and popular sites like this explicitly recommend using the oldest version possible.

[SorinGFS]

It has become increasingly difficult to push JSON Schema forward when large and popular sites like this explicitly recommend using the oldest version possible.

People use what they got. I don't know other option. If there was a proper alternative, schemastore would become obsolete overnight. I saw this problem long ago when I started my implementation project which mainly aims the schema management.