JsonValue is contravariant, should be covariant

[tamird]

Initial Checks

• I confirm that I'm using Pydantic V2

Description

See the code sample. It produces the following error in mypy:

foo.py:10: error: Incompatible types in assignment (expression has type "list[str]", target has type "JsonValue")  [assignment]

and in pyright:

Argument of type "list[str]" cannot be assigned to parameter "value" of type "JsonValue" in function "__setitem__"
  Type "list[str]" is incompatible with type "JsonValue"
    "list[str]" is incompatible with "List[JsonValue]"
      Type parameter "_T@list" is invariant, but "str" is not the same as "JsonValue"
      Consider switching from "list" to "Sequence" which is covariant
    "list[str]" is incompatible with "Dict[str, JsonValue]"
    "list[str]" is incompatible with "str"
    "list[str]" is incompatible with "bool"
    "list[str]" is incompatible with "int"

This works with an identical type where types.Dict is replaced with collections.abc.Mapping and types.List is replaced with collections.abc.Sequence.

Example Code

from pydantic import JsonValue

def bar() -> list[str]:
  raise NotImplementedError

foo: dict[str, JsonValue] = {
  "a": {"b": "hello"},
  "b": "c",
}
foo["l"] = bar()

Python, Pydantic & OS Version

poetry run python -c "import pydantic.version; print(pydantic.version.version_info())"
             pydantic version: 2.7.1
        pydantic-core version: 2.18.2
          pydantic-core build: profile=release pgo=true
                 install path: /Users/tamird/Library/Caches/pypoetry/virtualenvs/common-hf-Ms37h-py3.11/lib/python3.11/site-packages/pydantic
               python version: 3.11.9 (main, Apr 30 2024, 08:22:59) [Clang 15.0.0 (clang-1500.3.9.4)]
                     platform: macOS-14.4.1-arm64-arm-64bit
             related packages: mypy-1.10.0 typing_extensions-4.11.0 pydantic-extra-types-2.7.0 fastapi-0.110.3
                       commit: unknown

[sydney-runkle]

@tamird,

Thanks for reporting this - definitely looks like a typing bug. The concern I have here is that we can't use Sequence as the type hint, as not all sequence types are JSON serializable (consider sets, for example).

Here's one possible alternative:

if TYPE_CHECKING:
    # This seems to only be necessary for mypy
    J = TypeVar('J', bound='JsonValue')
    JsonValue: TypeAlias = Union[
        List[J],
        Dict[str, J],
        str,
        bool,
        int,
        float,
        None,
    ]

Let me know your thoughts on that. I can open a PR to fix this issue if you approve of said approach as well :).

[sydney-runkle]

One drawback is that this requires homogeneity in terms of the subtype of JsonValue used within a list or dict, which we might not want...

[tamird]

I'm pretty sure sets are not Sequences: https://mypy-play.net/?mypy=latest&python=3.12&gist=f71b466422216f85473dbb9880dd5f5c&flags=warn-unreachable.

What would be the problem with

    JsonValue: TypeAlias = Union[
        Sequence['JsonValue'],
        Mapping[str, 'JsonValue'],
        str,
        bool,
        int,
        float,
        None,
    ]

? Other than this discriminator having to be made more general.

[sydney-runkle]

@tamird,

Good point about sets. What about deque types? Deques are sequences, but aren't json serializable...

[tamird]

Can you give a runnable example? I don't understand why it isn't possible to serialize any sequence as a JSON array and to deserialize any JSON array as a list.