Improve annotations for various set methods

[AlexWaygood]

• Fixes some regressions introduced by Improve in-place BinOp methods for sets #7161 (see Annotations for set.__sub__ and related methods are inconsistent #9004 (comment)).
• Fixes Annotations for set.__sub__ and related methods are inconsistent #9004
• Fixes set[str].discard(str | None) #7121
• Fixes mypy doesn't understand set operations on optional mypy#13192 (cc. @finite-state-machine).
  • I wasn't initially sure that set[str | None] - set[None] came up often enough to be worth special-casing, but the activity on that issue has persuaded me otherwise.

It's a bit unfortunate how many type: ignores we have to introduce here, but I can't see a good way round them. I've added some test cases to make sure everything's working correctly.

[AlexWaygood]

Primer analysis:

• ibis is just an error message changing slightly

• werkzeug would just need to change their annotations on their subclass to match our new annotation for discard()

• The three pydantic hits are all new false positives caused by making set.__sub__ stricter.

  (You could argue that the third one is, strictly speaking, a true positive, in that it's working the way we "want" -- mypy is warning that there's a possibility AbstractSet[int] might be being subtracted from AbstractSet[str], which would be a no-op. But it doesn't seem like a "helpful" true positive. You'd have to do some more work to narrow the type before doing the subtraction, but it wouldn't really make your code any more type-safe.)

  These seem pretty unfortunate to me, and make me think that the better way of fixing Annotations for set.__sub__ and related methods are inconsistent #9004 is to make it so that __sub__, __isub__, difference and difference_update all accept AbstractSet[object] or Iterable[Any] rather than AbstractSet[_T | None]. @Akuli, thoughts on that?

  • https://github.com/pydantic/pydantic/blob/832225b90672c68e2d4067bd0ffecf834d62b48b/pydantic/v1/main.py#L901
  • https://github.com/pydantic/pydantic/blob/832225b90672c68e2d4067bd0ffecf834d62b48b/pydantic/main.py#L1257
  • https://github.com/pydantic/pydantic/blob/832225b90672c68e2d4067bd0ffecf834d62b48b/pydantic/deprecated/copy_internals.py#L222

• The new jax hit looks like a false positive, probably due to the fact that set.__sub__ is now an overloaded method. https://github.com/google/jax/blob/9352a00dab3c465a4e2c651cc6a0fa91e50ae7a2/jax/_src/lax/parallel.py#L512

[srittau]

This has conflicts now.

[AlexWaygood]

This has conflicts now.

I fixed the merge conflicts, but it's still not clear to me whether we want to fix the inconsistency between the different methods by making some annotations stricter (as this PR does), which seems to lead to false positives when running type checkers on user code. I think a better solution might be to make the methods consistent by making some annotations looser instead -- what do you think?

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

pydantic (https://github.com/samuelcolvin/pydantic)
+ pydantic/v1/main.py:901: error: Set comprehension has incompatible type Set[int | str]; expected Set[None]  [misc]
+ pydantic/main.py:1268: error: Argument 1 to "set" has incompatible type "AbstractSet[int] | AbstractSet[str] | Mapping[int, Any] | Mapping[str, Any]"; expected "Iterable[str | None]"  [arg-type]
+ pydantic/deprecated/copy_internals.py:222: error: Set comprehension has incompatible type Set[int | str]; expected Set[str | None]  [misc]

werkzeug (https://github.com/pallets/werkzeug)
+ src/werkzeug/datastructures/structures.pyi:197: error: Argument 1 of "discard" is incompatible with supertype "set"; supertype defines the argument type as "Optional[str]"  [override]
+ src/werkzeug/datastructures/structures.pyi:197: note: This violates the Liskov substitution principle
+ src/werkzeug/datastructures/structures.pyi:197: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#incompatible-overrides
+ src/werkzeug/datastructures/structures.pyi:197: error: Argument 1 of "discard" is incompatible with supertype "MutableSet"; supertype defines the argument type as "Optional[str]"  [override]

jax (https://github.com/google/jax)
+ jax/_src/lax/parallel.py:550: error: Need type annotation for "subs_batch" (hint: "subs_batch: Set[<type>] = ...")  [var-annotated]