Import modules/objects directly from v1 namespace

[exs-dwoodward]

Change Summary

Following on from this PR about adding a v1.py namespace to pydantic to better follow the v2 syntax of importing v1 objects, imports such as:

from pydantic.fields import ModelField
from pydantic.generics import GenericModel

or:

from pydantic import fields
from pydantic import generics

cannot be directly swapped for:

from pydantic.v1.fields import ModelField
from pydantic.v1.generics import GenericModel

or:

from pydantic.v1 import fields
from pydantic.v1 import generics

imports. If this was the case, this would make unpinning from v1 --> v2 much simpler as a simple find and replace would be all that is needed to convert from pydantic --> from pydantic.v1 instead (ofc still using v1 types and models).

One caveat with this approach is if this happens:

from pydantic.fields import ModelField
from pydantic.v1.fields import ModelField as ModelFieldFromV1

ModelField is ModelFieldFromV1 # False

but ideally the import syntax should be changed across a codebase rather than only in some places. Note that anything in the __init__.py pydantic namespace is still imported wholesale:

from pydantic import dataclasses
from pydantic.v1 import v1_dataclasses

dataclasses is v1_dataclasses # True

since these modules are imported directly from the underlying submodules. This also means that for the former import block, the ModelField in this case is imported twice.

Additionally, things like Pylance can't directly resolve the import path for pydantic.v1.fields imports using this syntax, but ideally that will be very transient since as soon as the find and replace is executed, any pydantic<2 pins should be able to be removed wholesale and the import will be resolved in pydantic v2.

Checklist

• The pull request title is a good summary of the changes - it will be used in the changelog
• Unit tests for the changes exist
• Tests pass on CI
• Documentation reflects the changes where applicable
• My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

Selected reviewer: @sydney-runkle

skip change file check

Fix #9357

Selected Reviewer: @PrettyWood

[exs-dwoodward]

This PR is ready for reviewing, please let me know if there are any other steps I can help out with here! Please review :)

[exs-dwoodward]

@PrettyWood is there anything else I need to do on my end on this PR? This is my first PR into pydantic so apologies if I've missed an automation somewhere!

[sydney-runkle]

@exs-dwoodward,

Great, thanks so much for your work! Apologies for the review delay - was focusing on FastUI for a bit. You've done everything perfectly here!

[sydney-runkle]

Great, thanks! We can release this with 1.10.16!

[sydney-runkle]

Working on getting the pipeline all green, will merge when that's done!

[sydney-runkle]

@exs-dwoodward,

Hmph, looks like tests are failing in some cases:

_______________________ test_can_import_modules_from_v1 ________________________
  
      def test_can_import_modules_from_v1() -> None:
          """That imports from any module in pydantic can be imported through
          ``pydantic.v1.<module>``"""
          for module_fname in os.listdir(pydantic.__path__[0]):
              if module_fname.startswith('_') or not module_fname.endswith('.py') or module_fname == 'v1.py':
                  continue
              module_name = module_fname[:-3]
      
  >           _ = importlib.import_module(f'pydantic.v1.{module_name}')
  
  /Users/runner/work/pydantic/pydantic/tests/test_v1.py:27: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/__init__.py:127: in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
  <frozen importlib._bootstrap>:1006: in _gcd_import
      ???
  <frozen importlib._bootstrap>:983: in _find_and_load
      ???
  <frozen importlib._bootstrap>:967: in _find_and_load_unlocked
      ???
  <frozen importlib._bootstrap>:677: in _load_unlocked
      ???
  <frozen importlib._bootstrap_external>:728: in exec_module
      ???
  <frozen importlib._bootstrap>:219: in _call_with_frames_removed
      ???
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  
      import sys
      from configparser import ConfigParser
      from typing import Any, Callable, Dict, List, Optional, Set, Tuple, Type as TypingType, Union
      
  >   from mypy.errorcodes import ErrorCode
  E   ModuleNotFoundError: No module named 'mypy'
  
  /Users/runner/work/pydantic/pydantic/pydantic/mypy.py:5: ModuleNotFoundError
  =========================== short test summary info ============================
  FAILED ../../../../../../../../../Users/runner/work/pydantic/pydantic/tests/test_v1.py::test_can_import_modules_from_v1

[sydney-runkle]

And another case:

https://github.com/pydantic/pydantic/actions/runs/8900758459/job/24443047366?pr=9162

[exs-dwoodward]

Thanks @sydney-runkle for looking through it!

hmm weird, running locally all my tests pass, though perhaps I should exclude the mypy module from being imported directly in the import test, given that mypy is an optional install.

I get the feeling that the ImportWarning's on the compiled linux tests seem to be due to the package spec not matching. I wonder if this is due to the relative imports used in pydantic (though I wouldn't want to make a massive change here). An alternative could be to employ a module level __getattr__ that imports the module asked for, but this feels quite heavy handed - wdyt?

[exs-dwoodward]

oh actually nevermind the __getattr__ can be pretty simple:

import pydantic

def __getattr__(name: str) -> Any:
     return getattr(pydantic, name)

seems to do the trick.

[sydney-runkle]

@exs-dwoodward,

Hmph, looks like we're still getting some test failures. I'd be ok with a more minimal approach here - not testing mypy or dotenv. Also, could you please fix the linting issue?

Thanks!

[exs-dwoodward]

I'm not entirely sure there is a nice fix for the linux compiled tests to be honest - not one that I can think of at least :( (that doesn't involve converting all the relative imports in pydantic to absolute) - given it only raises an ImportWarning - and that this 1.10.16 release would only be a temporary bound so that people can easily swap to pydantic 2 type imports, how worried are we about these warning failures? They only seem to crop up after the v1 tests are being run, so I'm wondering if swapping the test order would just magically make them disappear anyway.

[exs-dwoodward]

I would but I'm afraid that doing so might cause the linux-compiled tests to fail 😅 whereas swapping the relative --> absolute imports causes all the tests to pass (apart from the Windows ones)

[sydney-runkle]

Some context for my CI change, btw: https://github.com/pydantic/pydantic/pull/9305/files

I would but I'm afraid that doing so might cause the linux-compiled tests to fail 😅

Hmm I see what you mean... I'll take a closer look.

[sydney-runkle]

Also just as a side note - we can get this PR into v1.10.X-fixes, but this will probably be the last pydantic 1.10.X release, as we put a June 1st 2024 end date on maintenance 🚀.

[sydney-runkle]

I pinged @hramezani re these test failures, let's see what he thinks :).

[sydney-runkle]

@exs-dwoodward,

Hmm, so I got most of the tests passing. Let's revert those import changes and see if we can get the linux build tests passing :).

[sydney-runkle]

To your point, with the relative imports and the cibuildwheel pin: https://github.com/pydantic/pydantic/actions/runs/9351505506

[exs-dwoodward]

Yeah - looks like it might need the absolute imports - are you happy for me to revert the revert and use absolute imports?

[sydney-runkle]

@exs-dwoodward I'd like to get confirmation / explanation from @davidhewitt or @hramezani first, but I'll send an update soon!

[davidhewitt]

I don't have an immediate answer for what Cython might be doing to the code to break things when compiled; is it possible that the new __getattr__ implementation has the effect of making the wrong __package__ and __spec__ in that module?

[exs-dwoodward]

I believe that is exactly the issue - which appears to be 'fixed' by using absolute rather than relative imports.

If all the tests pass with absolute rather than relative imports - is there a particular reason other than caution to stick with relative?

[davidhewitt]

I believe that is exactly the issue - which appears to be 'fixed' by using absolute rather than relative imports.

I guess my worry is that if the relative imports broke things in a way we don't understand and the absolute imports will fixes them, is that just because stuff is still broken but we don't happen to be triggering the unhappy case?

What happens if we modify __getattr__ to exclude everything which starts with __ (maybe except for __all__ and __doc__)?

[exs-dwoodward]

Thats my only worry as I do not know exactly what cases pydantics CI tests cover - I can give that a go, see if it helps!

[exs-dwoodward]

I'm not too certain it will make a difference - since the modules / subpackages will still be importable, and it looks like their __package__ != __spec__.parent I believe, so this would stay the same. I'll try a commit anyways!

[davidhewitt]

Well the new error message seems interesting 👀

In PyO3 I think we've worked around similar errors by adding to sys.modules manually. E.g. sys.modules['pydantic.v1.fields'] = pydantic.fields.

Is this a good idea? I have no idea 😂

[davidhewitt]

Yeah so it looks like this hasn't really helped and is creating a lot of pain. Is the conclusion that the absolute imports was the only working option then? Maybe we just go with that and hope for the best...

[exs-dwoodward]

Okay nice - looks like the CI is all passing with the absolute imports in place now! 🚀

[sydney-runkle]

@exs-dwoodward,

Great work! We'll include this in the next v1.10.X patch release, then be done with V1 changes :). Thanks for all of your help (and patience) here!

Ping me anytime if you're looking for issues to contribute to, we really appreciate your work!

[sydney-runkle]

Note, due to this PR, we've made some updates to our V1 update logic in V2: #9639

I guess this was the problem with the absolute imports that we didn't think about...

[exs-dwoodward]

ah interesting. To be honest I would say that that is clearer anyway imo - interesting question though - is pydantic==1.10.16 version of the package expected to be present in the pydantic~=2 pydantic.v1 subdirectory anyway?

Given that this is simply to help with pydantic v1 --> v2 unpinning, I wouldn't have thought these changes needed baking into the v2 codebase anyway.

[phillipuniverse]

I'm very excited about this!! This is exactly what I need in my projects where I want to build in v1/v2 support ahead of time.

I think there's an issue here in mypy, here's an example of what I'm getting:

Checking mypy...
poetry run mypy .
shipwell_common_python/contrib/pydantic_v1/common.py:17: error: Cannot find
implementation or library stub for module named "pydantic.v1.generics"
[import-not-found]
    from pydantic.v1.generics import GenericModel
    ^
shipwell_common_python/contrib/pydantic_v1/common.py:17: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
shipwell_common_python/contrib/pydantic_v1/common.py:18: error: Cannot find
implementation or library stub for module named "pydantic.v1.json"
[import-not-found]
    from pydantic.v1.json import timedelta_isoformat
    ^
shipwell_common_python/contrib/pydantic_v1/common.py:52: error: Class cannot
subclass "GenericModel" (has type "Any")  [misc]
    class ShipwellBaseGenericSchema(GenericModel):
                                    ^
shipwell_common_python/contrib/pydantic_v1/schemas.py:5: error: Cannot find
implementation or library stub for module named "pydantic.v1.main"
[import-not-found]
    import pydantic.v1.main
    ^
shipwell_common_python/contrib/pydantic_v1/schemas.py:7: error: Cannot find
implementation or library stub for module named "pydantic.v1.utils"
[import-not-found]
    from pydantic.v1.utils import GetterDict
    ^
shipwell_common_python/contrib/pydantic_v1/schemas.py:14: error: Cannot find
implementation or library stub for module named "pydantic.v1.typing"
[import-not-found]
        from pydantic.v1.typing import DictStrAny

A workaround is to disable it, but then I lose Pydantic mypy features:

    [[tool.mypy.overrides]]
    module = [
        "pydantic.v1.*",
    ]
    ignore_missing_imports = true

Seems like the last part of this would be to update the mypy plugin as well to consider pydantic.v1.* alongside of pydantic.*? Or maybe just have a py.typed file in a first-class pydantic.v1 package? I'm not exactly sure the right way.

[sydney-runkle]

@exs-dwoodward,

We try to keep the latest version of v1 as a backport in v2, but we were able to update the related imports so that it still works, and I agree, things are more clear anyways!

@phillipuniverse,

Are you having this issue in V2 with the backport or V1?

[phillipuniverse]

@sydney-runkle sorry I should have clarified, this problem is with 1.10.16.

[exs-dwoodward]

I have had the same with mypy and essentially since I can now use the pin pydantic>=1.10.16,<3 just found and replaced from pydantic --> from pydantic.v1 then updated my environment that I type check with to use pydantic>1,<3 since that does resolve correctly to the .v1 subpackage.

Admittedly not the prettiest 'solution' but it means that whatever downstream packages can still functionally use the package with the swap, just the typing issues thrown by mypy will be pushed downwards until the downstream package does the same swap basically.

I'll try and have a think though how we could resolve the typing here - but ultimately I can't imagine it will be pretty given the somewhat cryptic way this imports from v1.py anyway.

[sydney-runkle]

@phillipuniverse @exs-dwoodward,

Thanks for the feedback. Happy to review any typing fixes here - that'd be much appreciated!

[phillipuniverse]

@exs-dwoodward I see, that's an interesting solve, and is workable although limited in correctness since you're not fully testing with exactly 1.10.16.

Here's an idea - what if we moved pydantic/v1.py -> pydantic/v1/__init.py__, and then added a py.typed next to it? Would that be a solve or wouldn't do anything because py.typed already exists in the pydantic package? Does something have to change in the mypy plugin do you think?