Improved Literal? types

[randolf-scholz]

This PR aims to formalize the behavior of Literal? types (Instance with last_known_value) by treating them as-if they were AnyOf-types (see python/typing#566).

See #19625 for a thorough specification.

For example, Literal["x"]? is treated as-if it were AnyOf[str, Literal["x"]].

• Fixes Specify behavior for Literal? types. #19625
• Fixes meet_types gives unexpected results when meeting literal and Instance #19560
• Fixes Regression: false positive avoid redundant cast with literal union #19055
• Fixes mypy cannot decide if a cast is necessary or redundant #21607
• Fixes Type Narrowing: if val == Literal should always narrow to Literal #16819
• Fixes Narrowing types to Literal using in syntax #12535
• Fixes Literal typeguard #10028
• Fixes TypedDict item insertion support #5359

New Tests

• JoinSuite.test_mixed_literal_types tests join between Literal? and other types.
• MeetSuite.test_mixed_literal_types tests meet between Literal? and other types
• TypeOpsSuite.test_simplified_union_with_mixed_str_literals2 tests simplified unions containing Literal? types
• SubtypingSuite.test_literal tests subtype checks with Literal? types
• RestrictionSuite: new test suite for testing the restrict_subtype_away method.
• testJoinLiteralInstanceAndEnum tests join between Literal? and StrEnum

[randolf-scholz]

Repro of the operator issue:

from typing import Literal, reveal_type

class Port:
    protocol: Literal["tcp", "udp", "icmp"]
    port: int | None

def show(*ports: int | Port) -> None:
    # master: set[tuple[L['tcp', 'udp', 'icmp'], int | None]]
    # PR:     set[tuple[L['tcp']?, int] | tuple[L['tcp', 'udp', 'icmp'], int | None]]
    reveal_type(  
        {         
            ( "tcp", port ) if isinstance(port, int) else ( port.protocol, port.port )
            for port in ports
        }
    )

On master the set comprehension combines both branches into set[tuple[L['tcp', 'udp', 'icmp'], int | None]].
With this PR, Literal['tcp']? is a subtype but not a proper subtype of Literal['tcp'], so the tuples are not simplified away in make_simplified_union, which checks proper subtyping. For context, here are other type checker results:

checker	result
mypy master	set[tuple[L['tcp', 'udp', 'icmp'], int | None]]
mypy PR	set[tuple[L['tcp']?, int] | tuple[L['tcp', 'udp', 'icmp'], int | None]]
pyright	set[tuple[str, int] | tuple[str, int | None]]
ty	set[tuple[str, int | None] | unknown]
pyrefly	set[tuple[str, int] | tuple[str, int | None]]

RFC: would this be considered a regression, or is the behavior proposed by the PR OK?

[randolf-scholz]

Maybe it's possible to do something smarter here, so that e.g. list[Literal["x"]] and list[Literal["x"]?] would be combined as well

[davidhalter]

Just in case you're interested, I have implicit literals implemented as well in Zuban. However it works in a different way, I store the implicit information on the literal. This makes it a bit easier to deal with literals in my opinion. I understand that that might cause a lot of changes in Mypy and it also might cause other issues, but I feel like it's a bit better to store it that way, because it essentially is a literal.

[randolf-scholz]

Update: The operator issue could be patched by a logical change in typeshed: python/typeshed#15160

[randolf-scholz]

Converting to draft since recent changes give "Non-overlapping equality check" errors.

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

spark (https://github.com/apache/spark)
+ python/pyspark/pandas/utils.py:757: error: Unused "type: ignore" comment  [unused-ignore]

steam.py (https://github.com/Gobot1234/steam.py)
- steam/ext/csgo/state.py:180: error: Argument "slot" to "Sticker" has incompatible type "int | None"; expected "Literal[0, 1, 2, 3, 4, 5] | None"  [arg-type]

prefect (https://github.com/PrefectHQ/prefect)
- src/prefect/concurrency/_asyncio.py:171: error: R? has no attribute "json"  [attr-defined]
+ src/prefect/task_worker.py:387: error: Argument 1 to "submit" of "Executor" has incompatible type "Callable[[Callable[_P, _T], **_P], _T]"; expected "Callable[[Callable[[Task[P, R], UUID | None, TaskRun | None, dict[str, Any] | None, PrefectFuture[Any] | Any | Iterable[PrefectFuture[Any] | Any] | None, Literal['state', 'result'], dict[str, set[RunInput]] | None, dict[str, Any] | None], R | State[Any] | None], Task[[VarArg(Any), KwArg(Any)], Any], UUID, TaskRun, dict[Any, Any], list[Any], str, Any | None], Any | State[Any] | None]"  [arg-type]

operator (https://github.com/canonical/operator)
- ops/pebble.py:1201: error: TypedDict key must be a string literal; expected one of ("command", "service-context", "environment", "user-id", "user", ...)  [literal-required]
- ops/pebble.py:1219: error: TypedDict key must be a string literal; expected one of ("url", "headers")  [literal-required]
+ ops/model.py:811: error: Incompatible types in assignment (expression has type "str", variable has type "Literal['tcp', 'udp', 'icmp']")  [assignment]

pyinstrument (https://github.com/joerick/pyinstrument)
- pyinstrument/context_manager.py:40: error: Argument 1 to "Profiler" has incompatible type "**dict[str, Literal['enabled', 'disabled', 'strict'] | float | bool | None]"; expected "float"  [arg-type]
+ pyinstrument/context_manager.py:40: error: Argument 1 to "Profiler" has incompatible type "**dict[str, Literal['enabled', 'strict'] | float | str | bool | None]"; expected "float"  [arg-type]
- pyinstrument/context_manager.py:40: error: Argument 1 to "Profiler" has incompatible type "**dict[str, Literal['enabled', 'disabled', 'strict'] | float | bool | None]"; expected "Literal['enabled', 'disabled', 'strict']"  [arg-type]
+ pyinstrument/context_manager.py:40: error: Argument 1 to "Profiler" has incompatible type "**dict[str, Literal['enabled', 'strict'] | float | str | bool | None]"; expected "Literal['enabled', 'disabled', 'strict']"  [arg-type]
- pyinstrument/context_manager.py:40: error: Argument 1 to "Profiler" has incompatible type "**dict[str, Literal['enabled', 'disabled', 'strict'] | float | bool | None]"; expected "bool | None"  [arg-type]
+ pyinstrument/context_manager.py:40: error: Argument 1 to "Profiler" has incompatible type "**dict[str, Literal['enabled', 'strict'] | float | str | bool | None]"; expected "bool | None"  [arg-type]

discord.py (https://github.com/Rapptz/discord.py)
- discord/app_commands/transformers.py:141: error: Incompatible types in assignment (expression has type "list[dict[str, Any]]", target has type "str | int")  [assignment]
+ discord/app_commands/transformers.py:141: error: Incompatible types in assignment (expression has type "list[dict[str, Any]]", target has type "bool | str | int")  [assignment]
- discord/app_commands/transformers.py:143: error: Incompatible types in assignment (expression has type "list[int]", target has type "str | int")  [assignment]
+ discord/app_commands/transformers.py:143: error: Incompatible types in assignment (expression has type "list[int]", target has type "bool | str | int")  [assignment]
- discord/app_commands/transformers.py:151: error: Incompatible types in assignment (expression has type "int | float", target has type "str | int")  [assignment]
+ discord/app_commands/transformers.py:151: error: Incompatible types in assignment (expression has type "int | float", target has type "bool | str | int")  [assignment]
- discord/app_commands/transformers.py:153: error: Incompatible types in assignment (expression has type "int | float", target has type "str | int")  [assignment]
+ discord/app_commands/transformers.py:153: error: Incompatible types in assignment (expression has type "int | float", target has type "bool | str | int")  [assignment]

pytest (https://github.com/pytest-dev/pytest)
+ testing/test_assertrewrite.py:963: error: Unused "type: ignore" comment  [unused-ignore]

dedupe (https://github.com/dedupeio/dedupe)
- dedupe/api.py:1547: error: Redundant cast to "Literal['match', 'distinct']"  [redundant-cast]

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/groupby/groupby.py:1609: error: Unused "type: ignore" comment  [unused-ignore]

materialize (https://github.com/MaterializeInc/materialize)
- misc/python/materialize/cli/scratch/__main__.py:95: error: Argument 1 to "join" of "str" has incompatible type "list[str | Callable[[Any], Any] | Callable[[], None] | Callable[[ArgumentParser], None] | Callable[[Namespace], None] | list[str]]"; expected "Iterable[str]"  [arg-type]
+ misc/python/materialize/cli/scratch/__main__.py:95: error: Argument 1 to "join" of "str" has incompatible type "list[Callable[[Any], Any] | Callable[[], None] | str | Callable[[ArgumentParser], None] | Callable[[Namespace], None] | list[str]]"; expected "Iterable[str]"  [arg-type]
- misc/python/materialize/cli/scratch/__main__.py:101: error: Argument 1 to "add_parser" of "_SubParsersAction" has incompatible type "str | Callable[[Any], Any] | Callable[[], None] | Callable[[ArgumentParser], None] | Callable[[Namespace], None] | list[str]"; expected "str"  [arg-type]
+ misc/python/materialize/cli/scratch/__main__.py:101: error: Argument 1 to "add_parser" of "_SubParsersAction" has incompatible type "Callable[[Any], Any] | Callable[[], None] | str | Callable[[ArgumentParser], None] | Callable[[Namespace], None] | list[str]"; expected "str"  [arg-type]
- misc/python/materialize/cli/scratch/__main__.py:101: error: Argument "aliases" to "add_parser" of "_SubParsersAction" has incompatible type "str | Callable[[Any], Any] | Callable[[], None] | Callable[[ArgumentParser], None] | Callable[[Namespace], None] | list[str]"; expected "Iterable[str]"  [arg-type]
+ misc/python/materialize/cli/scratch/__main__.py:101: error: Argument "aliases" to "add_parser" of "_SubParsersAction" has incompatible type "Callable[[Any], Any] | Callable[[], None] | str | Callable[[ArgumentParser], None] | Callable[[Namespace], None] | list[str]"; expected "Iterable[str]"  [arg-type]
- misc/python/materialize/cli/scratch/__main__.py:101: error: Argument "description" to "add_parser" of "_SubParsersAction" has incompatible type "str | Callable[[Any], Any] | Callable[[], None] | Callable[[ArgumentParser], None] | Callable[[Namespace], None] | list[str]"; expected "str | None"  [arg-type]
+ misc/python/materialize/cli/scratch/__main__.py:101: error: Argument "description" to "add_parser" of "_SubParsersAction" has incompatible type "Callable[[Any], Any] | Callable[[], None] | str | Callable[[ArgumentParser], None] | Callable[[Namespace], None] | list[str]"; expected "str | None"  [arg-type]
- misc/python/materialize/cli/scratch/__main__.py:101: error: Argument "help" to "add_parser" of "_SubParsersAction" has incompatible type "str | Callable[[Any], Any] | Callable[[], None] | Callable[[ArgumentParser], None] | Callable[[Namespace], None] | list[str]"; expected "str | None"  [arg-type]
+ misc/python/materialize/cli/scratch/__main__.py:101: error: Argument "help" to "add_parser" of "_SubParsersAction" has incompatible type "Callable[[Any], Any] | Callable[[], None] | str | Callable[[ArgumentParser], None] | Callable[[Namespace], None] | list[str]"; expected "str | None"  [arg-type]
- misc/python/materialize/cli/scratch/__main__.py:103: error: Argument 1 to "run" has incompatible type "ArgumentParser"; expected "Namespace"  [arg-type]
+ misc/python/materialize/cli/scratch/__main__.py:103: error: Argument 1 has incompatible type "ArgumentParser"; expected "Namespace"  [arg-type]

mitmproxy (https://github.com/mitmproxy/mitmproxy)
+ mitmproxy/net/server_spec.py:85: error: Unused "type: ignore" comment  [unused-ignore]
- mitmproxy/flowfilter.py:58: error: Redundant cast to "RegexFlag"  [redundant-cast]

jax (https://github.com/google/jax)
- jax/_src/pallas/mosaic/lowering.py:738: error: Redundant cast to "Literal['parallel', 'core_parallel', 'subcore_parallel', 'arbitrary']"  [redundant-cast]
- jax/_src/pallas/mosaic/lowering.py:915: error: Argument 1 to "_get_dimension_semantics" has incompatible type "tuple[Literal['parallel', 'core_parallel', 'subcore_parallel', 'arbitrary'] | GridDimensionSemantics, ...]"; expected "Sequence[str]"  [arg-type]
+ jax/_src/pallas/mosaic/lowering.py:915: error: Argument 1 to "_get_dimension_semantics" has incompatible type "tuple[Literal['core_parallel', 'subcore_parallel', 'arbitrary'] | GridDimensionSemantics | str, ...]"; expected "Sequence[str]"  [arg-type]
- jax/_src/internal_test_util/test_harnesses.py:2231: error: Incompatible types in assignment (expression has type "tuple[ndarray[tuple[Any, ...], dtype[unsignedinteger[_8Bit]]] | ndarray[tuple[Any, ...], dtype[unsignedinteger[_32Bit]]] | int, ...]", variable has type "list[list[int]]")  [assignment]
- jax/_src/internal_test_util/test_harnesses.py:2231: error: Incompatible types in assignment (expression has type "tuple[ndarray[tuple[Any, ...], dtype[unsignedinteger[_8Bit]]] | ndarray[tuple[Any, ...], dtype[unsignedinteger[_32Bit]]] | int, ...]", variable has type "tuple[int, ...]")  [assignment]
+ jax/_src/internal_test_util/test_harnesses.py:2231: error: Incompatible types in assignment (expression has type "tuple[object, ...]", variable has type "list[list[int]]")  [assignment]
+ jax/_src/internal_test_util/test_harnesses.py:2231: error: Incompatible types in assignment (expression has type "tuple[object, ...]", variable has type "tuple[int, ...]")  [assignment]
- jax/_src/internal_test_util/test_harnesses.py:2290: error: Incompatible types in assignment (expression has type "tuple[ndarray[tuple[Any, ...], dtype[unsignedinteger[_8Bit]]] | ndarray[tuple[Any, ...], dtype[unsignedinteger[_32Bit]]] | int, ...]", variable has type "list[list[int]]")  [assignment]
+ jax/_src/internal_test_util/test_harnesses.py:2290: error: Incompatible types in assignment (expression has type "tuple[int | ndarray[tuple[Any, ...], dtype[unsignedinteger[_8Bit]]] | ndarray[tuple[Any, ...], dtype[unsignedinteger[_32Bit]]], ...]", variable has type "list[list[int]]")  [assignment]

packaging (https://github.com/pypa/packaging)
+ src/packaging/version.py:297: error: Unused "type: ignore" comment  [unused-ignore]

meson (https://github.com/mesonbuild/meson)
- mesonbuild/modules/rust.py:890:60: error: Redundant cast to "Literal['c', 'cpp', 'cuda', 'fortran', 'd', 'objc', 'objcpp', 'rust', 'vala', 'cs', 'swift', 'java', 'cython', 'nasm', 'masm', 'linearasm']"  [redundant-cast]

sphinx (https://github.com/sphinx-doc/sphinx)
+ sphinx/ext/autodoc/_directive_options.py:225: error: Unused "type: ignore" comment  [unused-ignore]
+ sphinx/ext/autodoc/_directive_options.py:232: error: Unused "type: ignore" comment  [unused-ignore]

mongo-python-driver (https://github.com/mongodb/mongo-python-driver)
+ bson/__init__.py:1323: error: Cannot infer value of type parameter "_DocumentType" of "_bson_to_dict"  [misc]