typing.Self not supported in generic fields

[strangemonad]

Initial Checks

• I confirm that I'm using Pydantic V2

Description

It seems that with the recent introduction to support the Self type in pydantic 2.7 #5992, this doesn't cover support for generic fields where Self is the type var.

Alternatives:

1. For the code below, when making instantiate a new model, the FieldInfo likely needs to be updated. eg for the example below, Product has the following field info.

{'ref': FieldInfo(annotation=Ref[Self], required=True), 'name': FieldInfo(annotation=str, required=True)}

At the point of model creation, I believe it should be possible to update Ref[Self] to Ref[model_class]? The downside here is that the FieldInfo, while it reflects the reality of the semantics of the Self type, no longer matches the annotation verbatim making it harder to potentially debug, easier to introduce bugs (e.g. if the annotation value is used to look up the cached generic subclass) ad potentially incompatible with whatever direction the python core team takes __future__ annotations in.

2. Keep the existing field info and handle the case when building the field's core schema. This would mean a generic model subclass might need to be instantiated just in time and registered in then cache in the correct order for it to be visible to the core_schema machinery.

Example Code

from typing import TypeVar, Generic, Self
from pydantic import BaseModel

T = TypeVar("T", bound="Entity")


class Ref(BaseModel, Generic[T]):
    path: str


class Entity(BaseModel):
    ref: "Ref[Self]"


class Product(Entity):
    name: str

p = Product(name="test", ref=Ref[Product](path="f"))


```console
Traceback (most recent call last):
  File "/Users/shawn/Code/instance-bio/instance/apps/admin/src/pdg.py", line 18, in <module>
    p = Product(name="test", ref=Ref[Product](path="f"))
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/shawn/Code/instance-bio/instance/apps/admin/.venv/lib/python3.11/site-packages/pydantic/main.py", line 171, in __init__
    self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 1 validation error for Product
ref
  Input should be a valid dictionary or instance of Ref[Self] [type=model_type, input_value=Ref[Product](path='f'), input_type=Ref[Product]]
    For further information visit https://errors.pydantic.dev/2.6/v/model_type

### Python, Pydantic & OS Version

```Text
pydantic version: 2.7.1
        pydantic-core version: 2.18.2
          pydantic-core build: profile=release pgo=true
                 install path: /Users/shawn/Code/instance-bio/instance/libs/instance-firestore-entity/.venv/lib/python3.11/site-packages/pydantic
               python version: 3.11.4 (main, Aug 14 2023, 09:41:08) [Clang 14.0.3 (clang-1403.0.22.14.1)]
                     platform: macOS-14.4.1-arm64-arm-64bit
             related packages: typing_extensions-4.11.0 pyright-1.1.321
                       commit: unknown

[sydney-runkle]

@strangemonad,

Yep, you're right, this isn't currently supported. I've marked this as a feature request. PRs with support are welcome!

[NeevCohen]

@sydney-runkle Hey, I started looking into this, and it seems a little tricky to me.

In the definition of Entity

class Entity(BaseModel):
    ref: Ref[Self]

Ref's __class_getitem__ method gets called which is defined here, though when its called, the typevar_values parameter is just Self, without a way to access or determine what Self is actually referring to (none that I could tell at least).

My best idea was to perhaps look try to look up the stack to figure out what Self refers to, kind of like what super() does, though I'm not sure that's the best idea.

Do you perhaps have any ideas/directions into how this should be implemented?

[strangemonad]

@NeevCohen I don't believe it's a trial patch to handle the generic case. The code example above it's too hard (you use the new cls instance being constructed in the pydantic base model meta-class but e.g. what happens when Product has a subclass or what happens when there's an intermediate abstract class in the MRO / class hierarchy chain. There's also the issue that Self might occur in some arbitrarily complex type expression that needs to be interpreted. There's lots of edge cases to handle properly.

For this simple case, you can change Entity to have the following and it works for the main cases but it's not ideal

class Entity(BaseModel):
    ref: "Ref[Self]"

        @classmethod
    @override
    def __init_subclass__(cls, **kwargs):
        super().__init_subclass__(**kwargs)

        # Specialize the ref field type to the concrete Entity subclass.
        # This works around https://github.com/pydantic/pydantic/issues/9391
        # This patching of field-info must happen in __init_subclass__ (before
        # __pydantic_init_subclass__) so the correct core schema is generated for the model.
        ref_fi = cls.model_fields["ref"]
        # if ref_fi.annotation == Union[Ref[Self], None]:
        # but only if the type is not yet specialized so we can handle model lineages in a
        # slightly hacky way by setting the ref type to be the same across all versions.
        ref_fi.annotation = Union[Ref[cls], None]  # noqa: UP007  # we actually need to create a UnionType instance here.
        cls.model_fields["ref"] = ref_fi

@NeevCohen you could also look at BaseModel.model_construct and ModelMetaclass.__new__ to see where the new cls instance is created (you don't have to explicitly inspect the call stack).