[WIP] Make native parser the default

[ilevkivskyi]

For now the PR only contains updates for different error messages in the new parser. As I started working on making native parser the default, I discovered there are still several small bugs/inconsistencies.

I will start fixing them one by one in separate PRs, but you are also welcome to fix some if you have time, just let me know before you start working on them.

There is one question to discuss for now: it looks like there is no support in new parser for type comments in for and with statements. I vaguely remember this is intentional. What is our stance on this? Do we declare type comments for for and with deprecated and thus only supported by the old parser?

cc @JukkaL @hauntsaninja

[hauntsaninja]

Do we declare type comments for for and with deprecated and thus only supported by the old parser?

This seems totally fine to me. It doesn't affect public interfaces so there isn't going to be much fallout from losing these types.

[ilevkivskyi]

@JukkaL I completely forgot that some tests were already skipped with _no_native_parse. So in fact we have a bit more things to fix. Fortunately, it looks like all of them are easy. But this still means a bit more work is needed.

[ilevkivskyi]

(I will start with all the fixes that require changes in ast_serialize, and will handle the mypy-only things later)

[ilevkivskyi]

@JukkaL I have been looking into test failures, and one test case looks weird. I think the current behavior in testOverloadIfSplitFunctionDef (with the old parser) is wrong, while the behavior with the new parser is actually correct (note there is # flags: --always-true True so that behavior for f1 and f3 should be identical). Just wanted to double-check I am not missing something.

[ilevkivskyi]

@JukkaL Few more things for our discussion today:

• Ruff parser has a problem with parsing string that contain unicode surrogates, they all get replaced with the same replacement/placeholder symbol, see Surrogate code points are internally represented as U+FFFD REPLACEMENT CHARACTER astral-sh/ruff#13666. This is not a problem for mypy, but it is a problem for mypyc, as it affects runtime behavior, thus testUnicodeSurrogate fails. For some reasons fixing this is non-trivial, so RustPython (Rust-based Python interpreter that also uses Ruff parser) has an ~1000 lines patch that uses WTF, see Parse surrogates in string literals properly RustPython/RustPython#5629
• There are some cases where new parser errors are worse (either more verbose or more generic) than old parser errors. I am not going to spend much time on this, unless you think this is important.
• New parser does not error on some invalid function/variable type comments, and simply ignores them. I am not sure what to do about that, but I guess it may be fine.
• New parser ignores invalid # type: ignore comments instead of giving an error. I think this is an important thing to decide, potentially we may want to replicate old behavior exactly.
• New parser forcefully adds escapes even when they are not needed, for example b"what's up" is parsed as b"what\'s up". TBH I don't really like this, what is the motivation for this?

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

prefect (https://github.com/PrefectHQ/prefect)
- src/prefect/utilities/collections.py:504: error: "None" object is not iterable  [misc]
+ src/prefect/utilities/collections.py:505: error: "None" object is not iterable  [misc]
- src/prefect/_internal/concurrency/api.py:181: error: "Call" expects no type arguments, but 1 given  [type-arg]
- src/prefect/_internal/concurrency/api.py:181: error: Variable "prefect._internal.concurrency.api.T" is not valid as a type  [valid-type]
+ src/prefect/_internal/concurrency/api.py:182: error: Variable "prefect._internal.concurrency.api.T" is not valid as a type  [valid-type]
- src/prefect/_internal/concurrency/api.py:181: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
+ src/prefect/_internal/concurrency/api.py:182: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
+ src/prefect/_internal/concurrency/api.py:183: error: "Call" expects no type arguments, but 1 given  [type-arg]
+ src/prefect/_internal/concurrency/api.py:183: error: Variable "prefect._internal.concurrency.api.T" is not valid as a type  [valid-type]
+ src/prefect/_internal/concurrency/api.py:183: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
- src/prefect/server/api/deployments.py:463: error: Invalid "type: ignore" comment  [syntax]
+ src/prefect/server/api/deployments.py:463: error: Unused "type: ignore" comment  [unused-ignore]

operator (https://github.com/canonical/operator)
- ops/charm.py:137: error: Invalid "type: ignore" comment  [syntax]
- ops/charm.py:137: error: Incompatible types in assignment (expression has type "str | None", variable has type "str")  [assignment]

colour (https://github.com/colour-science/colour)
- colour/characterisation/aces_it.py:975: error: Ellipses cannot accompany other parameter types in function type signature  [syntax]
+ colour/characterisation/aces_it.py:976: error: Unexpected "..."  [misc]
+ colour/characterisation/aces_it.py:977: error: Unexpected "..."  [misc]

antidote (https://github.com/Finistere/antidote)
- src/antidote/lib/interface_ext/__init__.py:1359: error: "staticmethod" expects 2 type arguments, but 1 given  [type-arg]
+ src/antidote/lib/interface_ext/__init__.py:1361: error: "staticmethod" expects 2 type arguments, but 1 given  [type-arg]
- src/antidote/lib/interface_ext/__init__.py:1359: error: Missing type arguments for generic type "staticmethod"  [type-arg]
+ src/antidote/lib/interface_ext/__init__.py:1361: error: Missing type arguments for generic type "staticmethod"  [type-arg]

Tanjun (https://github.com/FasterSpeeding/Tanjun)
- tanjun/commands/menu.py:420: error: Type argument "_UserCallbackSigT" of "MessageCommand" must be a subtype of "def (MessageContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
+ tanjun/commands/menu.py:422: error: Type argument "_UserCallbackSigT" of "MessageCommand" must be a subtype of "def (MessageContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
- tanjun/commands/menu.py:420: error: Type argument "_UserCallbackSigT" of "SlashCommand" must be a subtype of "def (SlashContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
+ tanjun/commands/menu.py:422: error: Type argument "_UserCallbackSigT" of "SlashCommand" must be a subtype of "def (SlashContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
- tanjun/commands/menu.py:420: error: Type argument "_MessageCallbackSigT" of "MessageCommand" must be a subtype of "def (MessageContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
+ tanjun/commands/menu.py:424: error: Type argument "_MessageCallbackSigT" of "MessageCommand" must be a subtype of "def (MessageContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
- tanjun/commands/menu.py:420: error: Type argument "_MessageCallbackSigT" of "SlashCommand" must be a subtype of "def (SlashContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]
+ tanjun/commands/menu.py:424: error: Type argument "_MessageCallbackSigT" of "SlashCommand" must be a subtype of "def (SlashContext, /, *Any, **Any) -> Coroutine[Any, Any, None]"  [type-var]

trio (https://github.com/python-trio/trio)
+ src/trio/_ssl.py:230: error: Star unpack syntax: requires Python 3.11 or newer (current target: Python 3.10)  [syntax]
+ src/trio/_ssl.py:463: error: Star unpack syntax: requires Python 3.11 or newer (current target: Python 3.10)  [syntax]

ibis (https://github.com/ibis-project/ibis)
- ibis/selectors.py:434: error: Variable "ibis.common.collections.frozendict" is not valid as a type  [valid-type]
+ ibis/selectors.py:437: error: Variable "ibis.common.collections.frozendict" is not valid as a type  [valid-type]
- ibis/selectors.py:434: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
+ ibis/selectors.py:437: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases
- ibis/expr/types/generic.py:152: error: Function "ibis.expr.types.generic.Value.type" is not valid as a type  [valid-type]
+ ibis/expr/types/generic.py:164: error: Function "ibis.expr.types.generic.Value.type" is not valid as a type  [valid-type]
- ibis/expr/types/generic.py:152: note: Perhaps you need "Callable[...]" or a callback protocol?
+ ibis/expr/types/generic.py:164: note: Perhaps you need "Callable[...]" or a callback protocol?
- ibis/expr/types/generic.py:170: error: Function "ibis.expr.types.generic.Value.type" is not valid as a type  [valid-type]
+ ibis/expr/types/generic.py:171: error: Function "ibis.expr.types.generic.Value.type" is not valid as a type  [valid-type]
- ibis/expr/types/generic.py:170: note: Perhaps you need "Callable[...]" or a callback protocol?
+ ibis/expr/types/generic.py:171: note: Perhaps you need "Callable[...]" or a callback protocol?
- ibis/expr/types/temporal.py:428: error: "Value" expects no type arguments, but 1 given  [type-arg]
+ ibis/expr/types/temporal.py:431: error: "Value" expects no type arguments, but 1 given  [type-arg]
- ibis/expr/types/temporal.py:1161: error: "Value" expects no type arguments, but 1 given  [type-arg]
+ ibis/expr/types/temporal.py:1174: error: "Value" expects no type arguments, but 1 given  [type-arg]

static-frame (https://github.com/static-frame/static-frame)
+ static_frame/core/frame.py:7192: error: Unused "type: ignore" comment  [unused-ignore]
+ static_frame/core/frame.py:7193: error: Need type annotation for "name"  [var-annotated]
+ static_frame/core/frame.py:7435: error: Unused "type: ignore" comment  [unused-ignore]
+ static_frame/core/frame.py:7436: error: Need type annotation for "name"  [var-annotated]

apprise (https://github.com/caronc/apprise)
- apprise/plugins/fluxer.py:512: error: Unsupported operand types for + ("str" and "bool")  [operator]
+ apprise/plugins/fluxer.py:513: error: Unsupported operand types for + ("str" and "bool")  [operator]
- apprise/plugins/fluxer.py:512: note: Right operand is of type "bool | str"
+ apprise/plugins/fluxer.py:513: note: Right operand is of type "bool | str"
- apprise/plugins/fluxer.py:512: error: Incompatible types in assignment (expression has type "Any | str", target has type "bool")  [assignment]
+ apprise/plugins/fluxer.py:513: error: Incompatible types in assignment (expression has type "Any | str", target has type "bool")  [assignment]