Extracting vocabularies into the Feature Life Cycle

[gregsdennis]

Extracting vocbaularies is going to be a lot more work than extracting propertyDependencies was. Vocabulary language has been in the spec since 2019-09, and is very well integrated into the text.

Extraction

Much of it is casual mentions and can be replaced with "in this/that document" instead of "in the X vocabulary" and the like, but there are some hard parts.

Here is a quick survey of the occurrences of "vocbaulary" within the specs:

JSON Schema Core

• Section: Introduction
  • Mentions identifying vocab requirements.
• Section: Overview
  • Keywords are organized into vocabs.
  • Schema authors can make their own vocabs.
• Section: Instance Data Model
  • Directions for potential vocab authors to consider the data model rather than JSON text, with an example highlighting integer as a value type.
• Section: Non-JSON Instances
  • Allowance for vocab authors to support a superset of the data model.
• Section: Boolean JSON Schemas
  • Mentioned applicator vocab to clarify that the false schema is equivalent to { "not": {} }.
• Section: Schema Vocabularies
  • This whole section is about vocabs.
• Section: Meta-Schemas
  • Mentions that meta-schemas can validate keywords that aren't part of vocabs. (I suppose this is to cover inclusions like definitions, which has been retired.)
• Section: Identifiers
  • Core vocab mentioned as it defines $id and other identifier keywords.
  • A warning to vocab authors about defining other identifier keywords.
• Section: Assertions and Instance Primitive Types
  • maxLength and type are from the Validation vocab.
• Section: Reserved Locations
  • Mention that Core vocab defines location keywords.
• Section: The JSON Schema Core Vocabulary
  • "Vocabulary" language heavily used here. Will need a complete rewrite.
• Section: Meta-Schemas and Vocabularies
  • Describes interplay between the meta-schema and the vocabs that it declares.
  • Describes vocab schemas.
• Section: The $vocabulary Keyword
  • This whole section and its subsections.
• Section: Comments With $comment
  • Language that $comment is allowed anywhere vocab keywords are found.
• Section: Detecting a Meta-Schema
  • Mentions performing a vocab check on a meta-schema preload activity.
• Section: Compound Documents
  • Mentions each subschema could have varying vocab requirements.
• Section: References to Possible Non-Schemas
  • Mentions other vocab keywords as a possible location for subschemas.
• Section: A Vocabulary for Applying Subschemas
  • Applicators vocabulary. This section exists as it does because these keywords are grouped.
  • Defines the vocab, its URI, and its meta-schema URI.
• Section: Keyword Independence
  • Mentions the vocabulary.
• Section: items
  • Mentions the Unevaluated vocab.
• Section: properties
  • Mentions the Unevaluated vocab.
• Section: patternProperties
  • Mentions the Unevaluated vocab.
• Section: additionalProperties
  • Mentions the Unevaluated vocab.
• Section: contains
  • Mentions the Unevaluated vocab.
• Section: A Vocabulary for Unevaluated Locations
  • Unevaluated vocabulary. This section exists as it does because these keywords are grouped.
  • Defines the vocab, its URI, and its meta-schema URI.
• Section: Keyword Independence
  • Mentions that we're in a vocab.
• Section: Best practices for vocabulary and meta-schema authors
  • Vocab authorship.
• Section: Example meta-schema with vocabulary declarations
  • Meta-schema authorship with vocabs.

JSON Schema Validation

• JSON Schema Validation: A Vocabulary for Structural Validation of JSON
  • The document title.
• Section: Abstract
  • Mentions that the document specifies a vocab (not quite right since it actually specifies several vocabs).
• Section: Overview
  • Mentions that it specifies a meta-data vocab.
• Section: Meta-Schema
  • Mentions vocab URIs meta-schema (now "vocab schema") URIs.
• Section: A Vocabulary for Structural Validation
  • Defines URIs.
• Section: Vocabularies for Semantic Content With format
  • Defines URIs for both annotation and assertion vocabs.
• Section: Implementation Requirements
  • Indicates that the requirement depends on the vocab.
• Section: Format-Annotation Vocabulary
  • Editor note about specifying annotations but configuring for assertions.
• Section: Format-Assertion Vocabulary
  • Mentions of the vocab itself.
• Section: Custom format attributes
  • Mentions of about custom formats being specified in custom vocabs.
• Section: A Vocabulary for the Contents of String-Encoded Data ("Forward" subsection)
  • Defines URIs
• Section: A Vocabulary for Basic Meta-Data Annotations These general-purpose annotation
  • Defines URIs
• Section: Security Considerations
  • Not sure what this sentence is trying to say. I think it's just deference to Core for security stuff.
• Section: [Appendix] Keywords Moved from Validation to Core
  • Notes to vocab authors
• Section: [Appendix] ChangeLog
  • Historical stuff

I suspect that much of the changes will simply be reverting back to something like draft 7, before we introduced vocabs. There will be language and section changes (since some of the sections only serve to group keywords).

The proposal document is going to be large. I don't think there's a way around it. The discussion here is to figure out our strategy.

I've started threads below for each of the discussion points. Participation is open, and feel free to create threads for other (related) ideas.

Proposal Options

I have two orthogonal sets of ideas:

Vocab Organization

Options here are:

• Keep keywords (roughly) organized in vocabs the way they are.
• Re-organize keywords somehow.

Re-Integration vs New Documents

• Re-integrate the vocabs into the current Core & Validation documents. This is basically the current state.
• Further extract the keywords into vocab-specific documents, where each document defines the keywords that the vocab contains. This would leave just the Core vocab keywords in the Core document, and we'd have a set of vocab extension documents (e.g. Validation, Applicator, Content, etc.). (I think this is closer to an idea that @awwright was pushing some time ago.)

---

Update

I have created a draft PR to start this work.

[gregsdennis]

Discussion:

• Keep keywords (roughly) organized in vocabs the way they are.
• Re-organize keywords somehow.

[jdesrosiers]

I just don't see "composable building blocks for constructing dialects" as a valid argument here.

I don't know what the point of vocabularies is if not as composable building blocks for constructing dialects. Maybe you can elaborate a bit what you see as the purpose of vocabularies.

The only reason one might have to compose individual keywords is that they explicitly don't want a schema author to use a particular keyword. And even then, they can just create a meta-schema that includes a well-known vocab and my-unwanted-keyword: false to disallow it.

It's simple and it works, but it feels to me like a hack around the problem, not a solution.

I would like to point out that organizing the keywords by data type was the original keyword grouping before vocabs were introduced. (Ref: Draft 07 Validation)

I completely agree that that organization makes the most sense for the spec. I'd like to see the spec organized that way regardless of how we organize vocabularies. The vocabularies we define (if we define any at all), don't need to be coupled to how we define keywords in the spec.

(I'll read your proposal carefully and respond separately.)

[gregsdennis]

I recognize that the concepts in my comments above are not required for this action, so I'm going to hide them as off-topic. I will, however, be working toward finalizing the ideas into a more formal and complete proposal. I really think this can help the community, but proposing like this isn't the way.

[gregsdennis]

In the spirit of tabling my proposal, I think we should table keywords as the composable unit and keep vocabs as they are.

I'm happy to discuss your ideas elsewhere.

[gregsdennis]

Discussion:

• Re-integrate the vocabs into the current Core & Validation documents.
• Each vocab gets its own document.

[gregsdennis]

I think the only reason to define vocabularies in the spec is if the intention is that they are reusable components that others could use to construct their own dialects. We have a few that work well for that purpose like "unevaluated", "content", and "format-*". But there are others that really don't work well in that regard like "applicators", "validation", and "meta-data". They're too general and broad to be good components.

I think this pertains more to the other thread about re-organizing the vocabs. (I get that the point is that maybe we don't need to put the vocabs in the spec in the first place.)

[gregsdennis]

That means do the extraction keeping the current vocabularies and reorganize them later if we need to. That way this decision doesn't block the extraction step. - @jdesrosiers from the other thread

Yes, definitely. The first goal is to get vocabs out of the spec. For this, I'm really more concerned about how we build the proposal. Is the proposal just "let's update everything so it's exactly back to how it was in 2020-12?" or do we take more care with the initial document.

For example, we could have the proposal only add the sections that explicitly talk about how vocabs work and leave out all of the language integration stuff. I think it will be simpler this way, and once the feature is merged, then we can start integrating the language back into the rest of the documents in a way that fits.

[jdesrosiers]

Do we see this as a desire amongst our users? I haven't. The only reason to mess with vocabs at all is to create your own meta-schema. I've only seen a handful of those examples, and even in those cases, they were extending the current vocabulary set, not altering/constraining it (e.g. re-organizing or omitting keywords).

OpenAPI's 2.0 and 3.0 dialects of JSON Schema are examples. 2.0 is a subset that wouldn't fit along the lines of our vocabularies. 3.0 adds, removes, and modifies keywords. None of which can be handled the way we're doing our vocabularies. I vaguely remember there being a similar problem in the database vocab discussions. It also comes in IDL/DDL dialect discussions. Most of the validation keywords (with type being the critical exception) and some of the applicator vocab don't make sense in that domain.

It's possible that all of these cases could have been supported if we organized vocabs in a different way, but I think it's likely that there is no one set that will make sense in all domains.

I think it's an interesting idea; I just question whether it's needed, or even wanted, by by the community.

I think one of the strengths of this approach is that it's not just the most flexible thing we can do, it's also the simplest thing we can do. There's no big infrastructure we have to invent to support it. It's essentially the "do nothing" option. If no one in the community ends up constructing their own vocabularies, we haven't lost anything by making it possible.

[gregsdennis]

Vocabs were intended to allow for extension in a way that enabled the implementation to say whether the extension was supported. However, in a system where unknown keywords are forbidden, that behavior kind of comes automatically.

So do we still need vocabularies?

Identifiability

The design of vocabularies included defining a URI for each vocab that the implementation could recognize. If we just have keywords, then two parties could each define the same keyword to operate differently, and any schema that uses that keyword would behave incompatibly between them.

You could define that each keyword carries an identifier (and I recognize that HyperJump does), but you'd still have to work that into the meta-schema somehow to act as an indicator for the implementation to say, "Yes, I recognize the meaning of that keyword."

That's a valid solution, and I'm happy to explore it.

Implementation Support

Implementations aren't going to support each keyword in a separate package/module/whatever. It makes more sense to support and package groups of keywords. As a result, even if your custom meta-schema doesn't allow minLength, you're still going to get the support for it.

Extension keywords in particular are going to be defined in groups that serve a purpose, and (to my first point) implementations are more likely to support those keyword in those purposeful groups.

This doesn't mean that the core/default set of keywords (the ones published with the spec) needs to be split in any particular way, or at all. I expect the vast majority of implementations will support the full set as a single unit. However, I also want to consider other use cases (as you mentioned), like codegen which doesn't need the validation behavior, and find a way to start officially supporting those in a way that they don't feel they're partially implementing the spec. (That's what my idea rant was about, but I'll bring that up separately.)

But I think giving a way for implementations to say they support group X of keywords is valuable (as opposed to requiring them to say they support keywords X1, X2, etc.).

---

I expect there's some balance between these that we need to find, but I'm not sure what it is.

[jdesrosiers]

You could define that each keyword carries an identifier (and I recognize that HyperJump does), but you'd still have to work that into the meta-schema somehow to act as an indicator for the implementation to say, "Yes, I recognize the meaning of that keyword."

I'm not sure what you mean. The way I implemented it, the meta-schema is only concerned about the structure of the keyword, not it's semantics. Keywords, vocabs, and dialects are configured separately from meta-schemas no different than how the vocabulary system was designed to work.

addKeyword("https://json-schema.org/keyword/properties", /* keyword implementation */);
addVocabulary("https://json-schema.org/vocab/applicators", {
  "properties": "https://json-schema.org/keyword/properties",
  /* more keywords */
});

Then it determines what's supported the same way you would expect. It looks up the vocabularies declared in $vocabulary, and loads the keyword implementations for that vocabulary so they're available when they're encountered in a schema.

But I think giving a way for implementations to say they support group X of keywords is valuable (as opposed to requiring them to say they support keywords X1, X2, etc.).

I completely agree. I think the ability to define groups of keywords and treat them as a unit is a good idea. The thing I think is harmful is the coupling between keywords and vocabularies. I just want to break the coupling.

Keyword implementations can still be distributed as vocabulary modules, but dialect authors aren't required to use the full vocabulary if that doesn't make sense for them.

Giving keywords identifiers allows the DDL dialect author to define their own applicator vocabulary that excludes not and use that vocab in their dialect. Then implementations can include the module that provides all of the applicator keyword implementations. It will have an implementation for the not keyword, but since the custom vocabulary doesn't have not, it doesn't get loaded in that context. No need for a meta-schema hack. All the implementation needs is a keyword to keywordUri mapping for the vocabulary.

Even better would be if we had a vocabulary definition that the meta-schema could reference that contains the keyword => keywordURI mapping and then implementers don't need to provide that mapping as code. But, that's a completely different discussion for another time.

I expect there's some balance between these that we need to find, but I'm not sure what it is.

I think that what I'm describing is that balance. Vocabularies can be defined and distributed as a unit and that's probably how most people will use them. But, it's also trivially easy to construct your own vocabulary from those vocabulary modules if you need to.

[gregsdennis]

I found a breaking change when editing the Validation spec: format.

With vocabularies, a schema author has the power to specify whether format carries assertion behavior by using a meta-schema that requires the Format-Assertion Vocabulary.

However, without vocabularies, the schema author no longer has that power, and asserting format depends solely on implementation configuration.

[Relequestual]

While I appreciate we don't see every schema, I've never heard of anyone trying to do this.

[gregsdennis]

People consistently expect format to validate (per the spec, yes, but mainly just at all), despite the spec being clear that is just an annotation. They just don't understand that what they want is the assertion vocab.

[SorinGFS]

@gregsdennis

I really like your new way of thinking which aligns with many things that I have expressed in the past.

Here is a set of principles that I believe you should adhere to:

1. json-schema should be an environment where a json-schema keyword always means the same thing to everyone regardless of the context where it is used
2. the concept of json-schema dialect should be replaced by json-schema dictionary, which would mean the totality of json-schema keywords
3. once released, a json-schema keyword should never be modified, deprecated or removed
4. a json-schema keyword should only contain references to other json-schema keywords, never to $defs
5. the combination of several json-schema keywords is a json-schema vocabulary. A json-schema keyword can appear in many json-schema vocabularies.
6. a json-schema vocabulary can reference json-schema keywords and/or other json-schema vocabularies
7. a json-schema vocabulary is a context in which its elements respect the same set of rules described in its own specifications
8. json-schema schema users will refer to json-schema vocabularies instead of $schema, to which they will be able to add own keywords/vocabularies that they will describe in their own specifications. Without being recommended, they will be able to modify/remove/interpret differently the meaning of the keywords, and this will not involve the responsibility of json-schema in any way, but will be assumed by those who decide to refer to those vocabularies.
9. no longer needing $schema, the concept of draft is also replaced with vocabularies, and versioning also moves there. The vocabularies will evolve asynchronously, this means that there may be releases that involve only one vocabulary while the others remain with their unchanged version. E.g,: https://json-schema.org/vocabularies/applicator/v1.0 https://json-schema.org/vocabularies/validation/v2.0
10. having stable keywords json-schema may rejoin the IETF standardization path only with keywords part and by including vocabularies only as a concept for extension (without any mention about any use case).

Using this set of rules, json-schema should release new keywords only after long and lengthy analyses, and will be able to release vocabularies anytime without fear of breaking the functionality of already existing schemas.

The decentralized structure that I described above is also the structure used by schema.org. In this structure output, meta and schema would become vocabularies. Or I should say contexts ... not sure which one would be more descriptive for the job.

[gregsdennis]

A lot of your comments are actually changing how JSON Schema works in general. First, and again, this is not the place for that conversation. Second, we're not open to the kinds of sweeping changes you're describing at this time. We aim to keep functionality mostly as it is for the stable release, except that we need to extract features that aren't fully defined, like vocabularies.

[SorinGFS]

we're not open to the kinds of sweeping changes you're describing at this time.

I quickly went through the changes you made in that PR, I see that you are trying to adapt new needs with small steps, but I think a full restructuring and a full rewrite would be easier. And I think now is the time for such a thing. I'm sorry if this is off topic, but a consideration of the ideas above could make this topic a no topic at all.

[SorinGFS]

From what I see you are trying to extract the first part "$vocabulary" into another concern:

    "$vocabulary": {
        "https://json-schema.org/draft/next/vocab/core": true,
        "https://json-schema.org/draft/next/vocab/applicator": true,
        "https://json-schema.org/draft/next/vocab/unevaluated": true,
        "https://json-schema.org/draft/next/vocab/validation": true,
        "https://json-schema.org/draft/next/vocab/meta-data": true,
        "https://json-schema.org/draft/next/vocab/format-annotation": true,
        "https://json-schema.org/draft/next/vocab/content": true
    },
    "$dynamicAnchor": "meta",

    "title": "Core and Validation specifications meta-schema",
    "allOf": [
        {"$ref": "meta/core"},
        {"$ref": "meta/applicator"},
        {"$ref": "meta/unevaluated"},
        {"$ref": "meta/validation"},
        {"$ref": "meta/meta-data"},
        {"$ref": "meta/format-annotation"},
        {"$ref": "meta/content"}
    ],

...but if you have a distant look in fact those blocks "$vocabulary" and "allOf" are both vocabularies! The difference is that the second is the default json-schema` vocabulary, while the first is some kind of mechanism that allow someone to override the default. This is only needed because today there is no choice to refer a vocabulary directly!

[gregsdennis]

"$vocabulary" and "allOf" are both vocabularies! The difference is that the second is the default json-schema` vocabulary, while the first is some kind of mechanism that allow someone to override the default.

No, that's not how this works.

allOf works independently to provide validation by referencing each vocabulary schema, but those vocab schemas are not themselves the vocabularies.

$vocabulary works by listing the vocabulary URIs and indicating whether the vocabulary must be supported by the implementation in order to evaluate the schema.

They serve two completely different purposes. Yes, they appear similar, but they're quite different.

[SorinGFS]

As defined today, $vocabulary is something abstract, an alias for something that is actually as physical as possible. For example, the non-existent url "https://json-schema.org/draft/next/vocab/core" is an alias for the existing url "https://json-schema.org/draft/next/meta/core/schema"
If today we have to reference "$schema": "https://json-schema.org/draft/2020-12/schema" which includes everything contained in the specifications, what I suggest is that we should be able to reference just those parts that we need in a certain use case, e.g,: "https://json-schema.org/vocabularies/validation/v1.0"