Adding metadata schema to the code base itself

[ericspod]

Fixes #7303 #6959.

Description

This adds the schema file into the code base (but this maybe should be elsewhere). The changes implement a number of new things:

• Moved definitions into a $defs section per the JSON schema standard
• Permits multiple input arguments and return results from networks with arbitrary names using the patternProperties mechanism
• Allows the types of inputs and outputs to be, additional to just tensors, numbers, booleans, or strings
• Outputs after post processing can be specified with the post_processed_outputs section if they are significantly changed with the post-process transforms defined in scripts
• Multiple network IO formats can be specified in addition to network_data_format, these must follow the pattern <name>_data_format
• required_packages_version added in addition to optional_packages_version

#7253 depends on this schema change.

Types of changes

• Non-breaking change (fix or new feature that would not break existing functionality).
• Breaking change (fix or new feature that would cause existing functionality to change).
• New tests added to cover the changes.
• Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
• Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
• In-line docstrings updated.
• Documentation updated, tested make html command in the docs/ folder.

[ericspod]

Should this file exist here or put elsewhere? Currently the schema is stored in extra-test-data within a release, but it has to be somewhere accessible through a URL in the metadata.json files. #4048 references the issue of making this URL accessible.

[KumoLiu]

Should this file exist here or put elsewhere? Currently the schema is stored in extra-test-data within a release, but it has to be somewhere accessible through a URL in the metadata.json files. #4048 references the issue of making this URL accessible.

Yes, just make this draft PR to help review and leave comments. After finished we can move it in extra-test-data or somewhere else.

[yiheng-wang-nv]

Hi @ericspod , for existing bundles in model zoo, will this new schema impact them?

[ericspod]

I ran the schema for existing bundles, the ones that it failed on were the new generative bundles which didn't have a network_data_format and so are not compliant with the current schema anyway. The open question about requiring this field or not needs resolving here, if we don't make it required then all metadata.json files pass.

[KumoLiu]

My opinion on this one is that instead of renaming and replying on the pattern property, I think these two generative models missing the network_data_format, so we should add them. network_data_format should still be a required field.
https://github.com/Project-MONAI/model-zoo/blob/e99e61aeb6fd21fc8613fd6099463ca8d7420aca/models/brats_mri_axial_slices_generative_diffusion/configs/train_diffusion.json#L18C6-L18C17
What do you think?

[ericspod]

I think it should stay required so there's an assumption about which network is the "main" network for a bundle and can always be found. It should be a minor tweak to the generative bundles to make them conform.

[ericspod]

One thing that is different that we can easily add is that required_packages_version and optional_packages_version are both required. I think existing bundles should rename optional_packages_version to required_packages_version in their schemas if we want these properties to be required.

[yiheng-wang-nv]

Can we prepare a test case, like using 1) this schema and 2) prepare a fake metadata.json file to use verify_metadata in monai.bundle.scripts to verify that it works?

[KumoLiu]

Since we already know all of the packages we need after the bundle was created, we could remove 'optional_packages_version', which may be confusing, and instead, directly use 'required_packages_version'. What do you think?
cc @Nic-Ma @yiheng-wang-nv

[ericspod]

My idea was that required_packages_version is for additional packages this bundle absolutely required which wouldn't be present with a basic MONAI install, eg. nibabel which would be present only if the option was specified. optional_packages_version are ones which the bundle uses but doesn't strictly need.

[KumoLiu]

Yes, I know your points. But after bundle created, the creator needs to determine which packages are optional and then finished metadata.json. For users, even if the user knows that a package is optional, and they want to use the cheapest version, they still need to modify the config file themselves, so does an optional package still make sense for the bundle? Just wondering if it's necessary to complicate this, I'm open to any opinions.

[ericspod]

It's not necessary that the creator or the user has to modify anything. We have networks, transforms, and other components which check optional dependencies and will work fine if one isn't present. As an example, I'm working on a bundle which, in code in the scripts directory, optionally uses cupy but will resort to numpy if that isn't present.

[Nic-Ma]

We may need to clearly describe what the "optional package" means here to avoid confusion. I was also confused at the first look..

Thanks.

[ericspod]

I've updated the schema description line to be a bit clearer. We would also document in docs and wherever else relevant that the required packages are absolutely needed by the bundle, but the optional packages enable features or capabilities not strictly needed but may allow broader behaviours or faster performance. Eg. if an operation somewhere can use cupy to accelerate something but will default to numpy if this isn't present then cupy would be an optional package, or nibabel would be optional if the bundle can work with just numpy array loading and not need nifti loading.

[ericspod]

Can we prepare a test case, like using 1) this schema and 2) prepare a fake metadata.json file to use verify_metadata in monai.bundle.scripts to verify that it works?

Yes I can put that together shortly.

[ericspod]

Can we prepare a test case, like using 1) this schema and 2) prepare a fake metadata.json file to use verify_metadata in monai.bundle.scripts to verify that it works?

@yiheng-wang-nv I've added a notebook demonstrating this now. We won't want to merge this into dev, it's just a demo as the schema file is.

[ericspod]

Outstanding discussion points:

• Should required_packages_version and/or optional_packages_version be required at all? If required_packages_version is required then model zoo bundles will need updating.
• Should we allow multiple network formats with added [NAME]_data_format blocks but require there to always be a network_data_format block? The diffusion models in the zoo will need fixing if so.

[ericspod]

I've made required_packages_version mandatory but not optional_packages_version. We'll keep the [NAME]_data_format content of the schema. I've added documentation about other details as well.