[RFC] Archive stac-pydantic

[gadomski]

Summary

stac-pydantic is a Python package that provides pydantic models for STAC objects. Its main purpose is to provide:

• Serialization to and deserialization from JSON
• Validation

(De)serialization is also implemented in pystac, and validation is provided by the specification json schemas coupled with jsonschema validators, including the one in pystac. stac-pydantic doesn't provide any significant additional value to the ecosystem, and therefore should be archived to reduce "what package should I use" confusion.

Implementation

• Close all open PRs on stac-pydantic
• Update the readme to:
  • Point to this RFC
  • Show examples of how to make "vanilla" pydantic objects for STAC (as suggested in [RFC] Archive stac-pydantic #1252 (reply in thread))
• Archive the repo
• Announce the archival on the Gitter channel: https://app.gitter.im/#/room/#SpatioTemporal-Asset-Catalog_python:gitter.im
• Remove stac-pydantic dependency stac-utils/stac-fastapi#588

Drawbacks

Some libraries, most notably stac-fastapi, stac-fastapi-pgstac, and stac-fastapi-sqlalchemy, depend on stac-pydantic, though that dependency can be problematic (see stac-utils/stac-fastapi#588). While archiving stac-pydantic does not break any current software, since the PyPI releases won't change, it does mean that and dependent libraries will need to migrate away from stac-pydantic if they continue to evolve.

Rationale and alternatives

We could bring stac-pydantic up-to-date (e.g. via stac-utils/stac-pydantic#127), but that doesn't solve the problem of stac-pydantic and pystac overlapping considerably.

[thomas-maschler]

I believe the main benefit of this library is the nice integration with other libraries using Pydantic such as FastAPI including the libraries mentioned above. However, it is not a requirement.

When working on the PR using pydantic 2, I was hoping to get better performance out of stac-pydantic. After testing this now, this doesn't proof to be true. See screenshot of benchmark test, comparing

• Creating a STAC item using a Python dictionary without any type checks
• Creating a STAC item using Pystac
• Creating a STAC item using stac-pydantic (using branch from Pydantic 2 and STAC API 1.0 stac-utils/stac-pydantic#127)
  

As STAC evolves it will remain a burden to keep this library up to date. Since there is a significant overlap between this library and pystac and this library also seems to be much slower, archiving it might make sense.

[thomas-maschler]

import json

from pystac import Asset, Item, Link
from stac_pydantic import Item as PydanticItem

self_link_dict = {
    "rel": "self",
    "type": "application/geo+json",
    "href": "https://my.stac/v0/collections/mycollection/items/my_item",
}

collection_link_dict = {
    "rel": "collection",
    "type": "application/json",
    "href": "https://my.stac/v0/collections/mycollection",
}

parent_link_dict = {
    "rel": "parent",
    "type": "application/json",
    "href": "https://my.stac/v0/collections/mycollection",
}


item_dict = {
    "type": "Feature",
    "stac_version": "1.0.0",
    "id": "my_item",
    "collection": "mycollection",
    "geometry": {
        "coordinates": [
            [
                [149.19940003830305, -36.428645762574625],
                [149.05104951177714, -39.259123256881225],
                [152.81379005469256, -39.32599372427865],
                [152.82080633518694, -36.489064495233244],
                [149.19940003830305, -36.428645762574625],
            ]
        ],
        "type": "Polygon",
    },
    "bbox": [
        149.05104951177714,
        -39.32599372427865,
        152.82080633518694,
        -36.428645762574625,
    ],
    "properties": {
        "datetime": "2022-11-21T19:09:07.587000Z",
        "created": "2023-08-03T00:31:23.346946Z",
        "updated": "2023-08-03T00:31:23.346946Z",
    },
    "stac_extensions": [],
    "links": [self_link_dict, collection_link_dict, parent_link_dict],
    "assets": {
        "data": {
            "href": "https://my.stac/v0/collections/mycollection/items/my_item/data.tif",
            "title": "Data",
            "media_type": "image/tiff; application=geotiff",
            "roles": ["data"],
        },
        "thumbnail": {
            "href": "https://my.stac/v0/collections/mycollection/items/my_item/thumbnail.png",
            "media_type": "image/png",
            "roles": ["thumbnail"],
        },
        "metadata": {
            "href": "https://my.stac/v0/collections/mycollection/items/my_item/metadata.json",
            "media_type": "application/json",
            "roles": ["metadata"],
        },
    },
}

item_json = json.dumps(item_dict)

def item_from_dict() -> str:
    return json.dumps(item_dict)


def item_from_pydantic() -> str:
    return PydanticItem(**item_dict).model_dump_json()


def item_from_pystac() -> str:
    item = Item(
        id=item_dict["id"],
        geometry=item_dict["geometry"],
        bbox=item_dict["bbox"],
        datetime=item_dict["properties"]["datetime"],
        properties={
            "created": item_dict["properties"]["created"],
            "updated": item_dict["properties"]["updated"],
        },
    )

    data_asset = Asset(**item_dict["assets"]["data"])
    thumbnail_asset = Asset(**item_dict["assets"]["thumbnail"])
    metadata_asset = Asset(**item_dict["assets"]["metadata"])

    item.add_asset("data", data_asset)
    item.add_asset("thumbnail", thumbnail_asset)
    item.add_asset("metadata", metadata_asset)

    item.set_self_href(self_link_dict["href"])
    collection_link = Link(
        collection_link_dict["rel"],
        target=collection_link_dict["href"],
        media_type=collection_link_dict["type"],
    )
    parent_link = Link(
        parent_link_dict["rel"],
        target=parent_link_dict["href"],
        media_type=parent_link_dict["type"],
    )

    item.add_link(collection_link)
    item.add_link(parent_link)

    return json.dumps(item.to_dict())


def test_item_from_dict(benchmark):
    r = benchmark(item_from_dict)


def test_item_from_pydantic(benchmark):
    r = benchmark(item_from_pydantic)


def test_item_from_pystac(benchmark):
    r = benchmark(item_from_pystac)

[rbavery]

hi @eseglem did you discover anything related to pydantic performance? just checking since we are wondering if we should go with pydantic v2 (which feels like a cleaner and easier approach) or pystac for validation of the Label extension.

[eseglem]

@rbavery Thanks for the reminder. Hadn't got much of a chance to play with it. I went ahead and tried to split out the benchmarks a bit to have a more even comparison between them. I would probably go with pydantic myself, but depends a lot on use cases. Hopefully the following is useful info.

Using the current main branch of stac_pydantic (which is now using v2) and starting from the code provided above I did this: https://gist.github.com/eseglem/ddae063eefab545c122c93a2afc6cb86

• I added in orjson alongside json because I never use json anymore.
• Split apart TestDump and TestLoad sections to compare methods of input / output.
• Tried to show a variety of ways to do each to get a feel for it.

pytest test_stac_pydantic_benchmark.py -k TestDump

Of course orjson is faster than json, and dumping a dict is faster than the other items. While stac_pydantic has the highest min time, it also seems to be more consistent than pystac, with a much lower max time. And I think it ends up with a slightly better OPS because of that.

pytest test_stac_pydantic_benchmark.py -k TestLoad

Loading in a string to a dict with orjson of course is the fastest, no surprises there. Creating a pystac item directly, without parsing strings and everything already being correct types is also quite fast.

But I don't think it is a totally a fair comparison to what pydantic does. For example: pystac lets you have the datetime input be a string, and then breaks when you try to dump the item because its not an actual datetime. Using pydantic with validation basically eliminates that issue. And to be a more fair comparison I added in pydantic model_construct which skips all the validation logic. Some of the types are certainly wrong this way, but still an interesting number.

When you already have a dict, model_validate is 39.3k OPS vs the 48.8k OPS of pystac. While it is ~20% slower, you get the runtime type validation out of it. Which I think is huge if you are accepting user input. And it is a lot less code to handle based on the initial code.

I was a bit concerned by the speed of model_validate_json so I added a test to do model_validate(orjson.loads(...) and I don't really know what it is doing that makes it slower there. Appears that orjson is 15% faster than whatever pydantic is doing under the hood to handle the json string.

The numbers don't make total sense to me. I would expect that test_dict_orjson + test_pydantic_model_validate should be where test_pydantic_orjson_model_validate is, but somehow it is consistently significantly slower. And I have to wonder if it is somehow cheating when validating the same dict over and over. Maybe it's just that most of the types are already set correctly.

And it is very possible I am doing something totally wrong, but actually running validate on the pystac gets crushed by pydantic. I am wondering if it is downloading the spec every single time it tries to validate. Or if it is just that slow at validating against the spec. I didn't see an obvious way to cache the validator to make it faster, but hopefully someone else can chime in and improve that part.

For my particular use case, which is always accepting json from an untrusted end user, I want to have as much validation and safety as possible. So using pydantic is definitely the way to go. And based on what I see in these numbers I don't feel like any slowdowns from pydantic are all that bad.

Any thoughts @thomas-maschler ?

[eseglem]

Update: Sadly test_pydantic_model_validate was kind of cheating and modifying the datetime in place and not having to do the conversion each time. Which is why the numbers confused me. See: stac-utils/stac-pydantic#131

Using my branch here are the new numbers:

There is a slowdown in model_validate, so it is more like 37% slower and not 20% slower. However see a significant speedup in loading from json string. And the numbers are closer to what I expected.

I still feel its not all that bad. The times are in microseconds, so the difference in mean time is ~12 microseconds. In pydantic it is able to do it around 30k times per second. Which is plenty fast enough for my purposes at least. And totally worth it for all the benefits I see in pydantic.

[rbavery]

Thanks so much for this detailed analysis @eseglem. I'm happy with 37% correct validation of things like datetimes and will go ahead with pydantic.

[huard]

One use case for me is writing extensions for item and collection properties that enforce a strict controlled vocabulary, sometimes with hundreds of allowed values. Using pydantic, this can be done programmatically with under 100 lines of code. The equivalent json schema generated by pydantic has around 14,000 lines.

Also, real world metadata is often messy, and pydantic offers mechanisms to beat it into shape before feeding it into catalog items. In other words, pydantic does not only validate, it also parses, and this is really valuable.

All this to say that pydantic models might not be strictly necessary for vanilla STAC objects, but they are handy for extensions.

[gadomski]

All this to say that pydantic models might not be strictly necessary for vanilla STAC objects, but they are handy for extensions.

Right, but this library, as it currently exists, just provides the vanilla objects, which are relatively trivial to create for yourself if you're looking to build a more customized object schema. So I'm not sure stac-pydantic helps you very much?

[huard]

You're right, but over time, I could imagine that the pystac extensions would get their own pydantic models. If stac-pydantic disappears, then there's no recipe to follow.

I guess what I'm looking for is guidance on how to combine pystac and pydantic gracefully. What I've cobbled together so far feels clunky.

[gadomski]

If stac-pydantic disappears, then there's no recipe to follow.

It's a good point ... I wonder if this indicates that example pydantic models would be best in documentation, not in code. That way folks can't/won't depend on a possibly unmaintained library, but still have a recipie to follow.

What I've cobbled together so far feels clunky.

I'd say that doesn't mean you're doing it wrong -- original implementers of stac-pydantic found it awkward to get the validation logic Just Right™.

[geospatial-jeff]

It's a good point ... I wonder if this indicates that example pydantic models would be best in documentation, not in code.

An initial goals when writing stac-pydantic was just wrapping my head around the STAC spec. At the time the spec was less polished and I found it easier to think about STAC in terms of the pydantic models than what was actually defined in the spec. The spec has come a long ways and I don't think this is an issue anymore especially now that the spec is more stable.

You're right, but over time, I could imagine that the pystac extensions would get their own pydantic models.

At one point stac-pydantic did include STAC extensions but they were removed because they were pretty unwieldy and not that useful.

[keul]

Although I like the running discussion, and I've no real position to stac-pydantic keep-it-or-remove-it, I want to highlight that this is a blocker for moving onto pydantic 2.

Why is not possible to merge the fix for stac-pydantic, then later (after proper discussion that, at that point, can take months without bringing problems to nobody) it can be removed as a dependency, or not.

[thomas-maschler]

@keul you made a good point. Given FastAPI's strong dependency on Pydantic it seems rushed to deprecate/ archive the stac-pydantic library. We would need to see a strong proof of concept first to make sure that the current stac-fastapi* projects can work without the use of that library and someone's commitment to do the refactor. Otherwise these projects will be stuck with an old version of FastAPI/ Pydantic and won't be able to benefit from newer features.
Given that the major work to port stac-pydantic to Pydantic 2 is already done there is no reason to ditch that work at this point. I believe, the decision to archive the repo can be postponed until it is clear that no major actively maintained project still depends on it.

[gadomski]

Given FastAPI's strong dependency on Pydantic it seems rushed to deprecate/ archive the stac-pydantic library.

The only package that has a strong dependency is stac-fastapi-sqlalchemy. I opened an issue in July on stac-fastapi (stac-utils/stac-fastapi#588) advocating for its removal, and haven't seen any arguments against. stac-pydantic was in fact developed as a part of the original stac-fastapi development, but the developers realized it was not fit-for-purpose so mostly abandonded usage of stac-pydantic in stac-fastapi.

Given that the major work to port stac-pydantic to Pydantic 2 is already done there is no reason to ditch that work at this point.

It's a good point, and I appreciate that effort. It's true that archival of stac-pydantic and removal of stac-pydantic from stac-fastapi are two different, but related, issues.

If folks do find value in stac-pydantic, my preference would then be to strip down stac-pydantic considerably, making it only the pydantic models. All validation (IMO) should live in PySTAC, which has both json-schema validation and additional internal logic for cases that can't be caught by json-schema.

[thomas-maschler]

Thinking out loud:

stac-pydantic reimplements the JSON specifications as Pydantic models. It would be great if we were able to dynamically generate and extend such a model at runtime instead of having to define fields explicitly in code. This would made the package obsolete.

I imagine something like this:

Item = BaseModel.from_json_schema("https://raw.githubusercontent.com/radiantearth/stac-spec/v1.0.0/item-spec/json-schema/item.json")

Item.add_schema_extension("https://raw.githubusercontent.com/stac-extensions/raster/v1.1.0/json-schema/schema.json")

my_item = Item(id = ..., properties= {...}, ...)

Pydantic has a create_model class method and there have been discussions about adding a feature for dynamically creating models from jsonschema, but such implementation does not seem trivial.

[keul]

In this very interesting discussion you linked I see a reference to https://github.com/koxudaxi/datamodel-code-generator which is a tool I was not aware of 👀

[thomas-maschler]

That package doesn't generate models at runtime but rather writes them to file. It also has a very heavy dependency tree and tries to do a lot of things.

I gave it a shot and wrote a dynamic parser (still WIP, very barebone).
https://github.com/thomas-maschler/pydantic_jsonschema

I didn't add any validators yet and merging complex types might turn out to be tricky. I also only got it to work for Python >=3.11 so far. There is still quite a bit of work left to do, but I think it is possible.

[geospatial-jeff]

FWIW the original intent of stac-pydantic was to use create_model to dynamically generate models from either JSON schema or the stac_extensions member. Early versions of the code (before it was open sourced) did in fact use create_model but it never worked that well and was eventually removed in favor of explicitly defining each model.

[thomas-maschler]

Ok, that is interesting to know. I didn't get a chance to advance much on the repo I linked above, but I think given the improvements over the last years both to python and to pydantic, this seems to be more in reach now. Once I find the time, I will try to implement the full jsonschema spec.

[eseglem]

I have used datamodel-code-generator a fair bit. Based on my experience, I probably would not use it, or anything similar, in a totally automated / runtime fashion. It can be great for the bulk of the work, but I always had things I wanted to tweak. And, while it could be nice to do it all automatically at runtime, I think that could increase the complexity of the code and understanding what is happening. I personally would prefer to see all models as files so I can read or trace through them, set debug points, etc.

One of my projects has scripted use of the generator. It is mostly automated but I review any changes to the output file as a sanity check. Just to ensure nothing unexpected is happening. Which I think would be a risk of doing everything at runtime. Roughly the script does the following, and it should be possible to turn into a GitHub action that submits a PR if things change. Someone would still have to review / merge the changes, but the effort could be minimized. And everyone can have a better understanding of exactly how the models are defined with nothing unexpected happening at runtime.

• Pull the updated spec
• Run datamodel-codegen --output-model-type pydantic_v2.BaseModel (or custom BaseModel)
• Use a few regex to make some tweaks, based on my preferences:
  • I prefer to use Type Aliases over some things it does RootModel for.
  • Switch around a few things to use better built in types.
• Fix any formatting with black and ruff

Could be a possible middle ground, rather than entirely getting rid of it.