"""Update build by processing changes using fine-grained dependencies.

Use fine-grained dependencies to update targets in other modules that
may be affected by externally-visible changes in the changed modules.

This forms the core of the fine-grained incremental daemon mode. This
module is not used at all by the 'classic' (non-daemon) incremental
mode.

Here is some motivation for this mode:

* By keeping program state in memory between incremental runs, we
  only have to process changed modules, not their dependencies. The
  classic incremental mode has to deserialize the symbol tables of
  all dependencies of changed modules, which can be slow for large
  programs.

* Fine-grained dependencies allow processing only the relevant parts
  of modules indirectly affected by a change. Say, if only one function
  in a large module is affected by a change in another module, only this
  function is processed. The classic incremental mode always processes
  an entire file as a unit, which is typically much slower.

* It's possible to independently process individual modules within an
  import cycle (SCC). Small incremental changes can be fast independent
  of the size of the related SCC. In classic incremental mode, any change
  within a SCC requires the entire SCC to be processed, which can slow
  things down considerably.

Some terms:

* A *target* is a function/method definition or the top level of a module.
  We refer to targets using their fully qualified name (e.g.
  'mod.Cls.method'). Targets are the smallest units of processing during
  fine-grained incremental checking.

* A *trigger* represents the properties of a part of a program, and it
  gets triggered/fired when these properties change. For example,
  '<mod.func>' refers to a module-level function. It gets triggered if
  the signature of the function changes, or if the function is removed,
  for example.

Some program state is maintained across multiple build increments in
memory:

* The full ASTs of all modules are stored in memory all the time (this
  includes the type map).

* A fine-grained dependency map is maintained, which maps triggers to
  affected program locations (these can be targets, triggers, or
  classes). The latter determine what other parts of a program need to
  be processed again due to a fired trigger.

Here's a summary of how a fine-grained incremental program update happens:

* Determine which modules have changes in their source code since the
  previous update.

* Process changed modules one at a time. Perform a separate full update
  for each changed module, but only report the errors after all modules
  have been processed, since the intermediate states can generate bogus
  errors due to only seeing a partial set of changes.

* Each changed module is processed in full. We parse the module, and
  run semantic analysis to create a new AST and symbol table for the
  module. Reuse the existing ASTs and symbol tables of modules that
  have no changes in their source code. At the end of this stage, we have
  two ASTs and symbol tables for the changed module (the old and the new
  versions). The latter AST has not yet been type checked.

* Take a snapshot of the old symbol table. This is used later to determine
  which properties of the module have changed and which triggers to fire.

* Merge the old AST with the new AST, preserving the identities of
  externally visible AST nodes for which we can find a corresponding node
  in the new AST. (Look at mypy.server.astmerge for the details.) This
  way all external references to AST nodes in the changed module will
  continue to point to the right nodes (assuming they still have a valid
  target).

* Type check the new module.

* Take another snapshot of the symbol table of the changed module.
  Look at the differences between the old and new snapshots to determine
  which parts of the changed modules have changed. The result is a set of
  fired triggers.

* Using the dependency map and the fired triggers, decide which other
  targets have become stale and need to be reprocessed.

* Create new fine-grained dependencies for the changed module. We don't
  garbage collect old dependencies, since extra dependencies are relatively
  harmless (they take some memory and can theoretically slow things down
  a bit by causing redundant work). This is implemented in
  mypy.server.deps.

* Strip the stale AST nodes that we found above. This returns them to a
  state resembling the end of semantic analysis pass 1. We'll run semantic
  analysis again on the existing AST nodes, and since semantic analysis
  is not idempotent, we need to revert some changes made during semantic
  analysis. This is implemented in mypy.server.aststrip.

* Run semantic analyzer passes 2 and 3 on the stale AST nodes, and type
  check them. We also need to do the symbol table snapshot comparison
  dance to find any changes, and we need to merge ASTs to preserve AST node
  identities.

* If some triggers haven been fired, continue processing and repeat the
  previous steps until no triggers are fired.

This is module is tested using end-to-end fine-grained incremental mode
test cases (test-data/unit/fine-grained*.test).
"""

from __future__ import annotations

import os
import re
import sys
import time
from typing import Callable, Final, NamedTuple, Sequence, Union
from typing_extensions import TypeAlias as _TypeAlias

from mypy.build import (
    DEBUG_FINE_GRAINED,
    FAKE_ROOT_MODULE,
    BuildManager,
    BuildResult,
    Graph,
    State,
    load_graph,
    process_fresh_modules,
)
from mypy.checker import FineGrainedDeferredNode
from mypy.errors import CompileError
from mypy.fscache import FileSystemCache
from mypy.modulefinder import BuildSource
from mypy.nodes import (
    Decorator,
    FuncDef,
    ImportFrom,
    MypyFile,
    OverloadedFuncDef,
    SymbolNode,
    SymbolTable,
    TypeInfo,
)
from mypy.options import Options
from mypy.semanal_main import (
    core_modules,
    semantic_analysis_for_scc,
    semantic_analysis_for_targets,
)
from mypy.server.astdiff import (
    SymbolSnapshot,
    compare_symbol_table_snapshots,
    snapshot_symbol_table,
)
from mypy.server.astmerge import merge_asts
from mypy.server.aststrip import SavedAttributes, strip_target
from mypy.server.deps import get_dependencies_of_target, merge_dependencies
from mypy.server.target import trigger_to_target
from mypy.server.trigger import WILDCARD_TAG, make_trigger
from mypy.typestate import type_state
from mypy.util import module_prefix, split_target

MAX_ITER: Final = 1000

SENSITIVE_INTERNAL_MODULES = tuple(core_modules) + ("mypy_extensions", "typing_extensions")

class FineGrainedBuildManager:
    def __init__(self, result: BuildResult) -> None:
        """Initialize fine-grained build based on a batch build.

        Args:
            result: Result from the initialized build.
                    The manager and graph will be taken over by this class.
            manager: State of the build (mutated by this class)
            graph: Additional state of the build (mutated by this class)
        """
        manager = result.manager
        self.manager = manager
        self.graph = result.graph
        self.previous_modules = get_module_to_path_map(self.graph)
        self.deps = manager.fg_deps
        # Merge in any root dependencies that may not have been loaded
        merge_dependencies(manager.load_fine_grained_deps(FAKE_ROOT_MODULE), self.deps)
        self.previous_targets_with_errors = manager.errors.targets()
        self.previous_messages: list[str] = result.errors.copy()
        # Module, if any, that had blocking errors in the last run as (id, path) tuple.
        self.blocking_error: tuple[str, str] | None = None
        # Module that we haven't processed yet but that are known to be stale.
        self.stale: list[tuple[str, str]] = []
        # Disable the cache so that load_graph doesn't try going back to disk
        # for the cache.
        self.manager.cache_enabled = False

        # Some hints to the test suite about what is going on:
        # Active triggers during the last update
        self.triggered: list[str] = []
        # Modules passed to update during the last update
        self.changed_modules: list[tuple[str, str]] = []
        # Modules processed during the last update
        self.updated_modules: list[str] = []
        # Targets processed during last update (for testing only).
        self.processed_targets: list[str] = []

    def update(
        self,
        changed_modules: list[tuple[str, str]],
        removed_modules: list[tuple[str, str]],
        followed: bool = False,
    ) -> list[str]:
        """Update previous build result by processing changed modules.

        Also propagate changes to other modules as needed, but only process
        those parts of other modules that are affected by the changes. Retain
        the existing ASTs and symbol tables of unaffected modules.

        Reuses original BuildManager and Graph.

        Args:
            changed_modules: Modules changed since the previous update/build; each is
                a (module id, path) tuple. Includes modified and added modules.
                Assume this is correct; it's not validated here.
            removed_modules: Modules that have been deleted since the previous update
                or removed from the build.
            followed: If True, the modules were found through following imports

        Returns:
            A list of errors.
        """
        self.processed_targets.clear()
        changed_modules = changed_modules + removed_modules
        removed_set = {module for module, _ in removed_modules}
        self.changed_modules = changed_modules

        if not changed_modules:
            return self.previous_messages

        # Reset find_module's caches for the new build.
        self.manager.find_module_cache.clear()

        self.triggered = []
        self.updated_modules = []
        changed_modules = dedupe_modules(changed_modules + self.stale)
        initial_set = {id for id, _ in changed_modules}
        self.manager.log_fine_grained(
            "==== update %s ====" % ", ".join(repr(id) for id, _ in changed_modules)
        )
        if self.previous_targets_with_errors and is_verbose(self.manager):
            self.manager.log_fine_grained(
                "previous targets with errors: %s" % sorted(self.previous_targets_with_errors)
            )

        blocking_error = None
        if self.blocking_error:
            # Handle blocking errors first. We'll exit as soon as we find a
            # module that still has blocking errors.
            self.manager.log_fine_grained(f"existing blocker: {self.blocking_error[0]}")
            changed_modules = dedupe_modules([self.blocking_error] + changed_modules)
            blocking_error = self.blocking_error[0]
            self.blocking_error = None

        while True:
            result = self.update_one(
                changed_modules, initial_set, removed_set, blocking_error, followed
            )
            changed_modules, (next_id, next_path), blocker_messages = result

            if blocker_messages is not None:
                self.blocking_error = (next_id, next_path)
                self.stale = changed_modules
                messages = blocker_messages
                break

            # It looks like we are done processing everything, so now
            # reprocess all targets with errors. We are careful to
            # support the possibility that reprocessing an errored module
            # might trigger loading of a module, but I am not sure
            # if this can really happen.
            if not changed_modules:
                # N.B: We just checked next_id, so manager.errors contains
                # the errors from it. Thus we consider next_id up to date
                # when propagating changes from the errored targets,
                # which prevents us from reprocessing errors in it.
                changed_modules = propagate_changes_using_dependencies(
                    self.manager,
                    self.graph,
                    self.deps,
                    set(),
                    {next_id},
                    self.previous_targets_with_errors,
                    self.processed_targets,
                )
                changed_modules = dedupe_modules(changed_modules)
                if not changed_modules:
                    # Preserve state needed for the next update.
                    self.previous_targets_with_errors = self.manager.errors.targets()
                    messages = self.manager.errors.new_messages()
                    break

        messages = sort_messages_preserving_file_order(messages, self.previous_messages)
        self.previous_messages = messages.copy()
        return messages

    def trigger(self, target: str) -> list[str]:
        """Trigger a specific target explicitly.

        This is intended for use by the suggestions engine.
        """
        self.manager.errors.reset()
        changed_modules = propagate_changes_using_dependencies(
            self.manager,
            self.graph,
            self.deps,
            set(),
            set(),
            self.previous_targets_with_errors | {target},
            [],
        )
        # Preserve state needed for the next update.
        self.previous_targets_with_errors = self.manager.errors.targets()
        self.previous_messages = self.manager.errors.new_messages().copy()
        return self.update(changed_modules, [])

    def flush_cache(self) -> None:
        """Flush AST cache.

        This needs to be called after each increment, or file changes won't
        be detected reliably.
        """
        self.manager.ast_cache.clear()

    def update_one(
        self,
        changed_modules: list[tuple[str, str]],
        initial_set: set[str],
        removed_set: set[str],
        blocking_error: str | None,
        followed: bool,
    ) -> tuple[list[tuple[str, str]], tuple[str, str], list[str] | None]:
        """Process a module from the list of changed modules.

        Returns:
            Tuple with these items:

            - Updated list of pending changed modules as (module id, path) tuples
            - Module which was actually processed as (id, path) tuple
            - If there was a blocking error, the error messages from it
        """
        t0 = time.time()
        next_id, next_path = changed_modules.pop(0)

        # If we have a module with a blocking error that is no longer
        # in the import graph, we must skip it as otherwise we'll be
        # stuck with the blocking error.
        if (
            next_id == blocking_error
            and next_id not in self.previous_modules
            and next_id not in initial_set
        ):
            self.manager.log_fine_grained(
                f"skip {next_id!r} (module with blocking error not in import graph)"
            )
            return changed_modules, (next_id, next_path), None

        result = self.update_module(next_id, next_path, next_id in removed_set, followed)
        remaining, (next_id, next_path), blocker_messages = result
        changed_modules = [(id, path) for id, path in changed_modules if id != next_id]
        changed_modules = dedupe_modules(remaining + changed_modules)
        t1 = time.time()

        self.manager.log_fine_grained(
            f"update once: {next_id} in {t1 - t0:.3f}s - {len(changed_modules)} left"
        )

        return changed_modules, (next_id, next_path), blocker_messages

    def update_module(
        self, module: str, path: str, force_removed: bool, followed: bool
    ) -> tuple[list[tuple[str, str]], tuple[str, str], list[str] | None]:
        """Update a single modified module.

        If the module contains imports of previously unseen modules, only process one of
        the new modules and return the remaining work to be done.

        Args:
            module: Id of the module
            path: File system path of the module
            force_removed: If True, consider module removed from the build even if path
                exists (used for removing an existing file from the build)
            followed: Was this found via import following?

        Returns:
            Tuple with these items:

            - Remaining modules to process as (module id, path) tuples
            - Module which was actually processed as (id, path) tuple
            - If there was a blocking error, the error messages from it
        """
        self.manager.log_fine_grained(f"--- update single {module!r} ---")
        self.updated_modules.append(module)

        # builtins and friends could potentially get triggered because
        # of protocol stuff, but nothing good could possibly come from
        # actually updating them.
        if module in SENSITIVE_INTERNAL_MODULES:
            return [], (module, path), None

        manager = self.manager
        previous_modules = self.previous_modules
        graph = self.graph

        ensure_deps_loaded(module, self.deps, graph)

        # If this is an already existing module, make sure that we have
        # its tree loaded so that we can snapshot it for comparison.
        ensure_trees_loaded(manager, graph, [module])

        t0 = time.time()
        # Record symbol table snapshot of old version the changed module.
        old_snapshots: dict[str, dict[str, SymbolSnapshot]] = {}
        if module in manager.modules:
            snapshot = snapshot_symbol_table(module, manager.modules[module].names)
            old_snapshots[module] = snapshot

        manager.errors.reset()
        self.processed_targets.append(module)
        result = update_module_isolated(
            module, path, manager, previous_modules, graph, force_removed, followed
        )
        if isinstance(result, BlockedUpdate):
            # Blocking error -- just give up
            module, path, remaining, errors = result
            self.previous_modules = get_module_to_path_map(graph)
            return remaining, (module, path), errors
        assert isinstance(result, NormalUpdate)  # Work around #4124
        module, path, remaining, tree = result

        # TODO: What to do with stale dependencies?
        t1 = time.time()
        triggered = calculate_active_triggers(manager, old_snapshots, {module: tree})
        if is_verbose(self.manager):
            filtered = [trigger for trigger in triggered if not trigger.endswith("__>")]
            self.manager.log_fine_grained(f"triggered: {sorted(filtered)!r}")
        self.triggered.extend(triggered | self.previous_targets_with_errors)
        if module in graph:
            graph[module].update_fine_grained_deps(self.deps)
            graph[module].free_state()
        remaining += propagate_changes_using_dependencies(
            manager,
            graph,
            self.deps,
            triggered,
            {module},
            targets_with_errors=set(),
            processed_targets=self.processed_targets,
        )
        t2 = time.time()
        manager.add_stats(update_isolated_time=t1 - t0, propagate_time=t2 - t1)

        # Preserve state needed for the next update.
        self.previous_targets_with_errors.update(manager.errors.targets())
        self.previous_modules = get_module_to_path_map(graph)

        return remaining, (module, path), None

def find_unloaded_deps(
    manager: BuildManager, graph: dict[str, State], initial: Sequence[str]
) -> list[str]:
    """Find all the deps of the nodes in initial that haven't had their tree loaded.

    The key invariant here is that if a module is loaded, so are all
    of their dependencies. This means that when we encounter a loaded
    module, we don't need to explore its dependencies.  (This
    invariant is slightly violated when dependencies are added, which
    can be handled by calling find_unloaded_deps directly on the new
    dependencies.)
    """
    worklist = list(initial)
    seen: set[str] = set()
    unloaded = []
    while worklist:
        node = worklist.pop()
        if node in seen or node not in graph:
            continue
        seen.add(node)
        if node not in manager.modules:
            ancestors = graph[node].ancestors or []
            worklist.extend(graph[node].dependencies + ancestors)
            unloaded.append(node)

    return unloaded

def ensure_deps_loaded(module: str, deps: dict[str, set[str]], graph: dict[str, State]) -> None:
    """Ensure that the dependencies on a module are loaded.

    Dependencies are loaded into the 'deps' dictionary.

    This also requires loading dependencies from any parent modules,
    since dependencies will get stored with parent modules when a module
    doesn't exist.
    """
    if module in graph and graph[module].fine_grained_deps_loaded:
        return
    parts = module.split(".")
    for i in range(len(parts)):
        base = ".".join(parts[: i + 1])
        if base in graph and not graph[base].fine_grained_deps_loaded:
            merge_dependencies(graph[base].load_fine_grained_deps(), deps)
            graph[base].fine_grained_deps_loaded = True

def ensure_trees_loaded(
    manager: BuildManager, graph: dict[str, State], initial: Sequence[str]
) -> None:
    """Ensure that the modules in initial and their deps have loaded trees."""
    to_process = find_unloaded_deps(manager, graph, initial)
    if to_process:
        if is_verbose(manager):
            manager.log_fine_grained(
                "Calling process_fresh_modules on set of size {} ({})".format(
                    len(to_process), sorted(to_process)
                )
            )
        process_fresh_modules(graph, to_process, manager)

# The result of update_module_isolated when no blockers, with these items:
#
# - Id of the changed module (can be different from the module argument)
# - Path of the changed module
# - New AST for the changed module (None if module was deleted)
# - Remaining changed modules that are not processed yet as (module id, path)
#   tuples (non-empty if the original changed module imported other new
#   modules)
class NormalUpdate(NamedTuple):
    module: str
    path: str
    remaining: list[tuple[str, str]]
    tree: MypyFile | None

# The result of update_module_isolated when there is a blocking error. Items
# are similar to NormalUpdate (but there are fewer).
class BlockedUpdate(NamedTuple):
    module: str
    path: str
    remaining: list[tuple[str, str]]
    messages: list[str]

UpdateResult: _TypeAlias = Union[NormalUpdate, BlockedUpdate]

def update_module_isolated(
    module: str,
    path: str,
    manager: BuildManager,
    previous_modules: dict[str, str],
    graph: Graph,
    force_removed: bool,
    followed: bool,
) -> UpdateResult:
    """Build a new version of one changed module only.

    Don't propagate changes to elsewhere in the program. Raise CompileError on
    encountering a blocking error.

    Args:
        module: Changed module (modified, created or deleted)
        path: Path of the changed module
        manager: Build manager
        graph: Build graph
        force_removed: If True, consider the module removed from the build even it the
            file exists

    Returns a named tuple describing the result (see above for details).
    """
    if module not in graph:
        manager.log_fine_grained(f"new module {module!r}")

    if not manager.fscache.isfile(path) or force_removed:
        delete_module(module, path, graph, manager)
        return NormalUpdate(module, path, [], None)

    sources = get_sources(manager.fscache, previous_modules, [(module, path)], followed)

    if module in manager.missing_modules:
        manager.missing_modules.remove(module)

    orig_module = module
    orig_state = graph.get(module)
    orig_tree = manager.modules.get(module)

    def restore(ids: list[str]) -> None:
        # For each of the modules in ids, restore that id's old
        # manager.modules and graphs entries. (Except for the original
        # module, this means deleting them.)
        for id in ids:
            if id == orig_module and orig_tree:
                manager.modules[id] = orig_tree
            elif id in manager.modules:
                del manager.modules[id]
            if id == orig_module and orig_state:
                graph[id] = orig_state
            elif id in graph:
                del graph[id]

    new_modules: list[State] = []
    try:
        if module in graph:
            del graph[module]
        load_graph(sources, manager, graph, new_modules)
    except CompileError as err:
        # Parse error somewhere in the program -- a blocker
        assert err.module_with_blocker
        restore([module] + [st.id for st in new_modules])
        return BlockedUpdate(err.module_with_blocker, path, [], err.messages)

    # Reparsing the file may have brought in dependencies that we
    # didn't have before. Make sure that they are loaded to restore
    # the invariant that a module having a loaded tree implies that
    # its dependencies do as well.
    ensure_trees_loaded(manager, graph, graph[module].dependencies)

    # Find any other modules brought in by imports.
    changed_modules = [(st.id, st.xpath) for st in new_modules]

    # If there are multiple modules to process, only process one of them and return
    # the remaining ones to the caller.
    if len(changed_modules) > 1:
        # As an optimization, look for a module that imports no other changed modules.
        module, path = find_relative_leaf_module(changed_modules, graph)
        changed_modules.remove((module, path))
        remaining_modules = changed_modules
        # The remaining modules haven't been processed yet so drop them.
        restore([id for id, _ in remaining_modules])
        manager.log_fine_grained(f"--> {module!r} (newly imported)")
    else:
        remaining_modules = []

    state = graph[module]

    # Process the changed file.
    state.parse_file()
    assert state.tree is not None, "file must be at least parsed"
    t0 = time.time()
    try:
        semantic_analysis_for_scc(graph, [state.id], manager.errors)
    except CompileError as err:
        # There was a blocking error, so module AST is incomplete. Restore old modules.
        restore([module])
        return BlockedUpdate(module, path, remaining_modules, err.messages)

    # Merge old and new ASTs.
    new_modules_dict: dict[str, MypyFile | None] = {module: state.tree}
    replace_modules_with_new_variants(manager, graph, {orig_module: orig_tree}, new_modules_dict)

    t1 = time.time()
    # Perform type checking.
    state.type_checker().reset()
    state.type_check_first_pass()
    state.type_check_second_pass()
    state.detect_possibly_undefined_vars()
    t2 = time.time()
    state.finish_passes()
    t3 = time.time()
    manager.add_stats(semanal_time=t1 - t0, typecheck_time=t2 - t1, finish_passes_time=t3 - t2)

    graph[module] = state

    return NormalUpdate(module, path, remaining_modules, state.tree)

def find_relative_leaf_module(modules: list[tuple[str, str]], graph: Graph) -> tuple[str, str]:
    """Find a module in a list that directly imports no other module in the list.

    If no such module exists, return the lexicographically first module from the list.
    Always return one of the items in the modules list.

    NOTE: If both 'abc' and 'typing' have changed, an effect of the above rule is that
        we prefer 'abc', even if both are in the same SCC. This works around a false
        positive in 'typing', at least in tests.

    Args:
        modules: List of (module, path) tuples (non-empty)
        graph: Program import graph that contains all modules in the module list
    """
    assert modules
    # Sort for repeatable results.
    modules = sorted(modules)
    module_set = {module for module, _ in modules}
    for module, path in modules:
        state = graph[module]
        if len(set(state.dependencies) & module_set) == 0:
            # Found it!
            return module, path
    # Could not find any. Just return the first module (by lexicographic order).
    return modules[0]

def delete_module(module_id: str, path: str, graph: Graph, manager: BuildManager) -> None:
    manager.log_fine_grained(f"delete module {module_id!r}")
    # TODO: Remove deps for the module (this only affects memory use, not correctness)
    if module_id in graph:
        del graph[module_id]
    if module_id in manager.modules:
        del manager.modules[module_id]
    components = module_id.split(".")
    if len(components) > 1:
        # Delete reference to module in parent module.
        parent_id = ".".join(components[:-1])
        # If parent module is ignored, it won't be included in the modules dictionary.
        if parent_id in manager.modules:
            parent = manager.modules[parent_id]
            if components[-1] in parent.names:
                del parent.names[components[-1]]
    # If the module is removed from the build but still exists, then
    # we mark it as missing so that it will get picked up by import from still.
    if manager.fscache.isfile(path):
        manager.missing_modules.add(module_id)

def dedupe_modules(modules: list[tuple[str, str]]) -> list[tuple[str, str]]:
    seen: set[str] = set()
    result = []
    for id, path in modules:
        if id not in seen:
            seen.add(id)
            result.append((id, path))
    return result

def get_module_to_path_map(graph: Graph) -> dict[str, str]:
    return {module: node.xpath for module, node in graph.items()}

def get_sources(
    fscache: FileSystemCache,
    modules: dict[str, str],
    changed_modules: list[tuple[str, str]],
    followed: bool,
) -> list[BuildSource]:
    sources = []
    for id, path in changed_modules:
        if fscache.isfile(path):
            sources.append(BuildSource(path, id, None, followed=followed))
    return sources

def calculate_active_triggers(
    manager: BuildManager,
    old_snapshots: dict[str, dict[str, SymbolSnapshot]],
    new_modules: dict[str, MypyFile | None],
) -> set[str]:
    """Determine activated triggers by comparing old and new symbol tables.

    For example, if only the signature of function m.f is different in the new
    symbol table, return {'<m.f>'}.
    """
    names: set[str] = set()
    for id in new_modules:
        snapshot1 = old_snapshots.get(id)
        if snapshot1 is None:
            names.add(id)
            snapshot1 = {}
        new = new_modules[id]
        if new is None:
            snapshot2 = snapshot_symbol_table(id, SymbolTable())
            names.add(id)
        else:
            snapshot2 = snapshot_symbol_table(id, new.names)
        diff = compare_symbol_table_snapshots(id, snapshot1, snapshot2)
        package_nesting_level = id.count(".")
        for item in diff.copy():
            if item.count(".") <= package_nesting_level + 1 and item.split(".")[-1] not in (
                "__builtins__",
                "__file__",
                "__name__",
                "__package__",
                "__doc__",
            ):
                # Activate catch-all wildcard trigger for top-level module changes (used for
                # "from m import *"). This also gets triggered by changes to module-private
                # entries, but as these unneeded dependencies only result in extra processing,
                # it's a minor problem.
                #
                # TODO: Some __* names cause mistriggers. Fix the underlying issue instead of
                #     special casing them here.
                diff.add(id + WILDCARD_TAG)
            if item.count(".") > package_nesting_level + 1:
                # These are for changes within classes, used by protocols.
                diff.add(item.rsplit(".", 1)[0] + WILDCARD_TAG)

        names |= diff
    return {make_trigger(name) for name in names}

def replace_modules_with_new_variants(
    manager: BuildManager,
    graph: dict[str, State],
    old_modules: dict[str, MypyFile | None],
    new_modules: dict[str, MypyFile | None],
) -> None:
    """Replace modules with newly builds versions.

    Retain the identities of externally visible AST nodes in the
    old ASTs so that references to the affected modules from other
    modules will still be valid (unless something was deleted or
    replaced with an incompatible definition, in which case there
    will be dangling references that will be handled by
    propagate_changes_using_dependencies).
    """
    for id in new_modules:
        preserved_module = old_modules.get(id)
        new_module = new_modules[id]
        if preserved_module and new_module is not None:
            merge_asts(preserved_module, preserved_module.names, new_module, new_module.names)
            manager.modules[id] = preserved_module
            graph[id].tree = preserved_module

def propagate_changes_using_dependencies(
    manager: BuildManager,
    graph: dict[str, State],
    deps: dict[str, set[str]],
    triggered: set[str],
    up_to_date_modules: set[str],
    targets_with_errors: set[str],
    processed_targets: list[str],
) -> list[tuple[str, str]]:
    """Transitively rechecks targets based on triggers and the dependency map.

    Returns a list (module id, path) tuples representing modules that contain
    a target that needs to be reprocessed but that has not been parsed yet.

    Processed targets should be appended to processed_targets (used in tests only,
    to test the order of processing targets).
    """

    num_iter = 0
    remaining_modules: list[tuple[str, str]] = []

    # Propagate changes until nothing visible has changed during the last
    # iteration.
    while triggered or targets_with_errors:
        num_iter += 1
        if num_iter > MAX_ITER:
            raise RuntimeError("Max number of iterations (%d) reached (endless loop?)" % MAX_ITER)

        todo, unloaded, stale_protos = find_targets_recursive(
            manager, graph, triggered, deps, up_to_date_modules
        )
        # TODO: we sort to make it deterministic, but this is *incredibly* ad hoc
        remaining_modules.extend((id, graph[id].xpath) for id in sorted(unloaded))
        # Also process targets that used to have errors, as otherwise some
        # errors might be lost.
        for target in targets_with_errors:
            id = module_prefix(graph, target)
            if id is not None and id not in up_to_date_modules:
                if id not in todo:
                    todo[id] = set()
                manager.log_fine_grained(f"process target with error: {target}")
                more_nodes, _ = lookup_target(manager, target)
                todo[id].update(more_nodes)
        triggered = set()
        # First invalidate subtype caches in all stale protocols.
        # We need to do this to avoid false negatives if the protocol itself is
        # unchanged, but was marked stale because its sub- (or super-) type changed.
        for info in stale_protos:
            type_state.reset_subtype_caches_for(info)
        # Then fully reprocess all targets.
        # TODO: Preserve order (set is not optimal)
        for id, nodes in sorted(todo.items(), key=lambda x: x[0]):
            assert id not in up_to_date_modules
            triggered |= reprocess_nodes(manager, graph, id, nodes, deps, processed_targets)
        # Changes elsewhere may require us to reprocess modules that were
        # previously considered up to date. For example, there may be a
        # dependency loop that loops back to an originally processed module.
        up_to_date_modules = set()
        targets_with_errors = set()
        if is_verbose(manager):
            manager.log_fine_grained(f"triggered: {list(triggered)!r}")

    return remaining_modules

def find_targets_recursive(
    manager: BuildManager,
    graph: Graph,
    triggers: set[str],
    deps: dict[str, set[str]],
    up_to_date_modules: set[str],
) -> tuple[dict[str, set[FineGrainedDeferredNode]], set[str], set[TypeInfo]]:
    """Find names of all targets that need to reprocessed, given some triggers.

    Returns: A tuple containing a:
     * Dictionary from module id to a set of stale targets.
     * A set of module ids for unparsed modules with stale targets.
    """
    result: dict[str, set[FineGrainedDeferredNode]] = {}
    worklist = triggers
    processed: set[str] = set()
    stale_protos: set[TypeInfo] = set()
    unloaded_files: set[str] = set()

    # Find AST nodes corresponding to each target.
    #
    # TODO: Don't rely on a set, since the items are in an unpredictable order.
    while worklist:
        processed |= worklist
        current = worklist
        worklist = set()
        for target in current:
            if target.startswith("<"):
                module_id = module_prefix(graph, trigger_to_target(target))
                if module_id:
                    ensure_deps_loaded(module_id, deps, graph)

                worklist |= deps.get(target, set()) - processed
            else:
                module_id = module_prefix(graph, target)
                if module_id is None:
                    # Deleted module.
                    continue
                if module_id in up_to_date_modules:
                    # Already processed.
                    continue
                if (
                    module_id not in manager.modules
                    or manager.modules[module_id].is_cache_skeleton
                ):
                    # We haven't actually parsed and checked the module, so we don't have
                    # access to the actual nodes.
                    # Add it to the queue of files that need to be processed fully.
                    unloaded_files.add(module_id)
                    continue

                if module_id not in result:
                    result[module_id] = set()
                manager.log_fine_grained(f"process: {target}")
                deferred, stale_proto = lookup_target(manager, target)
                if stale_proto:
                    stale_protos.add(stale_proto)
                result[module_id].update(deferred)

    return result, unloaded_files, stale_protos

def reprocess_nodes(
    manager: BuildManager,
    graph: dict[str, State],
    module_id: str,
    nodeset: set[FineGrainedDeferredNode],
    deps: dict[str, set[str]],
    processed_targets: list[str],
) -> set[str]:
    """Reprocess a set of nodes within a single module.

    Return fired triggers.
    """
    if module_id not in graph:
        manager.log_fine_grained("%s not in graph (blocking errors or deleted?)" % module_id)
        return set()

    file_node = manager.modules[module_id]
    old_symbols = find_symbol_tables_recursive(file_node.fullname, file_node.names)
    old_symbols = {name: names.copy() for name, names in old_symbols.items()}
    old_symbols_snapshot = snapshot_symbol_table(file_node.fullname, file_node.names)

    def key(node: FineGrainedDeferredNode) -> int:
        # Unlike modules which are sorted by name within SCC,
        # nodes within the same module are sorted by line number, because
        # this is how they are processed in normal mode.
        return node.node.line

    nodes = sorted(nodeset, key=key)

    state = graph[module_id]
    options = state.options
    manager.errors.set_file_ignored_lines(
        file_node.path, file_node.ignored_lines, options.ignore_errors or state.ignore_all
    )
    manager.errors.set_skipped_lines(file_node.path, file_node.skipped_lines)

    targets = set()
    for node in nodes:
        target = target_from_node(module_id, node.node)
        if target is not None:
            targets.add(target)
    manager.errors.clear_errors_in_targets(file_node.path, targets)

    # If one of the nodes is the module itself, emit any errors that
    # happened before semantic analysis.
    for target in targets:
        if target == module_id:
            for info in graph[module_id].early_errors:
                manager.errors.add_error_info(info)

    # Strip semantic analysis information.
    saved_attrs: SavedAttributes = {}
    for deferred in nodes:
        processed_targets.append(deferred.node.fullname)
        strip_target(deferred.node, saved_attrs)
    semantic_analysis_for_targets(graph[module_id], nodes, graph, saved_attrs)
    # Merge symbol tables to preserve identities of AST nodes. The file node will remain
    # the same, but other nodes may have been recreated with different identities, such as
    # NamedTuples defined using assignment statements.
    new_symbols = find_symbol_tables_recursive(file_node.fullname, file_node.names)
    for name in old_symbols:
        if name in new_symbols:
            merge_asts(file_node, old_symbols[name], file_node, new_symbols[name])

    # Type check.
    checker = graph[module_id].type_checker()
    checker.reset()
    # We seem to need additional passes in fine-grained incremental mode.
    checker.pass_num = 0
    checker.last_pass = 3
    more = checker.check_second_pass(nodes)
    while more:
        more = False
        if graph[module_id].type_checker().check_second_pass():
            more = True

    if manager.options.export_types:
        manager.all_types.update(graph[module_id].type_map())

    new_symbols_snapshot = snapshot_symbol_table(file_node.fullname, file_node.names)
    # Check if any attribute types were changed and need to be propagated further.
    changed = compare_symbol_table_snapshots(
        file_node.fullname, old_symbols_snapshot, new_symbols_snapshot
    )
    new_triggered = {make_trigger(name) for name in changed}

    # Dependencies may have changed.
    update_deps(module_id, nodes, graph, deps, options)

    # Report missing imports.
    graph[module_id].verify_dependencies()

    graph[module_id].free_state()

    return new_triggered

def find_symbol_tables_recursive(prefix: str, symbols: SymbolTable) -> dict[str, SymbolTable]:
    """Find all nested symbol tables.

    Args:
        prefix: Full name prefix (used for return value keys and to filter result so that
            cross references to other modules aren't included)
        symbols: Root symbol table

    Returns a dictionary from full name to corresponding symbol table.
    """
    result = {}
    result[prefix] = symbols
    for name, node in symbols.items():
        if isinstance(node.node, TypeInfo) and node.node.fullname.startswith(prefix + "."):
            more = find_symbol_tables_recursive(prefix + "." + name, node.node.names)
            result.update(more)
    return result

def update_deps(
    module_id: str,
    nodes: list[FineGrainedDeferredNode],
    graph: dict[str, State],
    deps: dict[str, set[str]],
    options: Options,
) -> None:
    for deferred in nodes:
        node = deferred.node
        type_map = graph[module_id].type_map()
        tree = graph[module_id].tree
        assert tree is not None, "Tree must be processed at this stage"
        new_deps = get_dependencies_of_target(
            module_id, tree, node, type_map, options.python_version
        )
        for trigger, targets in new_deps.items():
            deps.setdefault(trigger, set()).update(targets)
    # Merge also the newly added protocol deps (if any).
    type_state.update_protocol_deps(deps)

def lookup_target(
    manager: BuildManager, target: str
) -> tuple[list[FineGrainedDeferredNode], TypeInfo | None]:
    """Look up a target by fully-qualified name.

    The first item in the return tuple is a list of deferred nodes that
    needs to be reprocessed. If the target represents a TypeInfo corresponding
    to a protocol, return it as a second item in the return tuple, otherwise None.
    """

    def not_found() -> None:
        manager.log_fine_grained(f"Can't find matching target for {target} (stale dependency?)")

    modules = manager.modules
    items = split_target(modules, target)
    if items is None:
        not_found()  # Stale dependency
        return [], None
    module, rest = items
    if rest:
        components = rest.split(".")
    else:
        components = []
    node: SymbolNode | None = modules[module]
    file: MypyFile | None = None
    active_class = None
    for c in components:
        if isinstance(node, TypeInfo):
            active_class = node
        if isinstance(node, MypyFile):
            file = node
        if not isinstance(node, (MypyFile, TypeInfo)) or c not in node.names:
            not_found()  # Stale dependency
            return [], None
        # Don't reprocess plugin generated targets. They should get
        # stripped and regenerated when the containing target is
        # reprocessed.
        if node.names[c].plugin_generated:
            return [], None
        node = node.names[c].node
    if isinstance(node, TypeInfo):
        # A ClassDef target covers the body of the class and everything defined
        # within it.  To get the body we include the entire surrounding target,
        # typically a module top-level, since we don't support processing class
        # bodies as separate entities for simplicity.
        assert file is not None
        if node.fullname != target:
            # This is a reference to a different TypeInfo, likely due to a stale dependency.
            # Processing them would spell trouble -- for example, we could be refreshing
            # a deserialized TypeInfo with missing attributes.
            not_found()
            return [], None
        result = [FineGrainedDeferredNode(file, None)]
        stale_info: TypeInfo | None = None
        if node.is_protocol:
            stale_info = node
        for name, symnode in node.names.items():
            node = symnode.node
            if isinstance(node, FuncDef):
                method, _ = lookup_target(manager, target + "." + name)
                result.extend(method)
        return result, stale_info
    if isinstance(node, Decorator):
        # Decorator targets actually refer to the function definition only.
        node = node.func
    if not isinstance(node, (FuncDef, MypyFile, OverloadedFuncDef)):
        # The target can't be refreshed. It's possible that the target was
        # changed to another type and we have a stale dependency pointing to it.
        not_found()
        return [], None
    if node.fullname != target:
        # Stale reference points to something unexpected. We shouldn't process since the
        # context will be wrong and it could be a partially initialized deserialized node.
        not_found()
        return [], None
    return [FineGrainedDeferredNode(node, active_class)], None

def is_verbose(manager: BuildManager) -> bool:
    return manager.options.verbosity >= 1 or DEBUG_FINE_GRAINED

def target_from_node(module: str, node: FuncDef | MypyFile | OverloadedFuncDef) -> str | None:
    """Return the target name corresponding to a deferred node.

    Args:
        module: Must be module id of the module that defines 'node'

    Returns the target name, or None if the node is not a valid target in the given
    module (for example, if it's actually defined in another module).
    """
    if isinstance(node, MypyFile):
        if module != node.fullname:
            # Actually a reference to another module -- likely a stale dependency.
            return None
        return module
    else:  # OverloadedFuncDef or FuncDef
        if node.info:
            return f"{node.info.fullname}.{node.name}"
        else:
            return f"{module}.{node.name}"

if sys.platform != "win32":
    INIT_SUFFIXES: Final = ("/__init__.py", "/__init__.pyi")
else:
    INIT_SUFFIXES: Final = (
        os.sep + "__init__.py",
        os.sep + "__init__.pyi",
        os.altsep + "__init__.py",
        os.altsep + "__init__.pyi",
    )

def refresh_suppressed_submodules(
    module: str,
    path: str | None,
    deps: dict[str, set[str]],
    graph: Graph,
    fscache: FileSystemCache,
    refresh_file: Callable[[str, str], list[str]],
) -> list[str] | None:
    """Look for submodules that are now suppressed in target package.

    If a submodule a.b gets added, we need to mark it as suppressed
    in modules that contain "from a import b". Previously we assumed
    that 'a.b' is not a module but a regular name.

    This is only relevant when following imports normally.

    Args:
        module: target package in which to look for submodules
        path: path of the module
        refresh_file: function that reads the AST of a module (returns error messages)

    Return a list of errors from refresh_file() if it was called. If the
    return value is None, we didn't call refresh_file().
    """
    messages = None
    if path is None or not path.endswith(INIT_SUFFIXES):
        # Only packages have submodules.
        return None
    # Find any submodules present in the directory.
    pkgdir = os.path.dirname(path)
    try:
        entries = fscache.listdir(pkgdir)
    except FileNotFoundError:
        entries = []
    for fnam in entries:
        if (
            not fnam.endswith((".py", ".pyi"))
            or fnam.startswith("__init__.")
            or fnam.count(".") != 1
        ):
            continue
        shortname = fnam.split(".")[0]
        submodule = module + "." + shortname
        trigger = make_trigger(submodule)

        # We may be missing the required fine-grained deps.
        ensure_deps_loaded(module, deps, graph)

        if trigger in deps:
            for dep in deps[trigger]:
                # We can ignore <...> deps since a submodule can't trigger any.
                state = graph.get(dep)
                if not state:
                    # Maybe it's a non-top-level target. We only care about the module.
                    dep_module = module_prefix(graph, dep)
                    if dep_module is not None:
                        state = graph.get(dep_module)
                if state:
                    # Is the file may missing an AST in case it's read from cache?
                    if state.tree is None:
                        # Create AST for the file. This may produce some new errors
                        # that we need to propagate.
                        assert state.path is not None
                        messages = refresh_file(state.id, state.path)
                    tree = state.tree
                    assert tree  # Will be fine, due to refresh_file() above
                    for imp in tree.imports:
                        if isinstance(imp, ImportFrom):
                            if (
                                imp.id == module
                                and any(name == shortname for name, _ in imp.names)
                                and submodule not in state.suppressed_set
                            ):
                                state.suppressed.append(submodule)
                                state.suppressed_set.add(submodule)
    return messages

def extract_fnam_from_message(message: str) -> str | None:
    m = re.match(r"([^:]+):[0-9]+: (error|note): ", message)
    if m:
        return m.group(1)
    return None

def extract_possible_fnam_from_message(message: str) -> str:
    # This may return non-path things if there is some random colon on the line
    return message.split(":", 1)[0]

def sort_messages_preserving_file_order(
    messages: list[str], prev_messages: list[str]
) -> list[str]:
    """Sort messages so that the order of files is preserved.

    An update generates messages so that the files can be in a fairly
    arbitrary order.  Preserve the order of files to avoid messages
    getting reshuffled continuously.  If there are messages in
    additional files, sort them towards the end.
    """
    # Calculate file order from the previous messages
    n = 0
    order = {}
    for msg in prev_messages:
        fnam = extract_fnam_from_message(msg)
        if fnam and fnam not in order:
            order[fnam] = n
            n += 1

    # Related messages must be sorted as a group of successive lines
    groups = []
    i = 0
    while i < len(messages):
        msg = messages[i]
        maybe_fnam = extract_possible_fnam_from_message(msg)
        group = [msg]
        if maybe_fnam in order:
            # This looks like a file name. Collect all lines related to this message.
            while (
                i + 1 < len(messages)
                and extract_possible_fnam_from_message(messages[i + 1]) not in order
                and extract_fnam_from_message(messages[i + 1]) is None
                and not messages[i + 1].startswith("mypy: ")
            ):
                i += 1
                group.append(messages[i])
        groups.append((order.get(maybe_fnam, n), group))
        i += 1

    groups = sorted(groups, key=lambda g: g[0])
    result = []
    for key, group in groups:
        result.extend(group)
    return result


from __future__ import annotations

from typing import Final

"""Shared logic between our three mypy parser files."""

_NON_BINARY_MAGIC_METHODS: Final = {
    "__abs__",
    "__call__",
    "__complex__",
    "__contains__",
    "__del__",
    "__delattr__",
    "__delitem__",
    "__enter__",
    "__exit__",
    "__float__",
    "__getattr__",
    "__getattribute__",
    "__getitem__",
    "__hex__",
    "__init__",
    "__init_subclass__",
    "__int__",
    "__invert__",
    "__iter__",
    "__len__",
    "__long__",
    "__neg__",
    "__new__",
    "__oct__",
    "__pos__",
    "__repr__",
    "__reversed__",
    "__setattr__",
    "__setitem__",
    "__str__",
}

MAGIC_METHODS_ALLOWING_KWARGS: Final = {
    "__init__",
    "__init_subclass__",
    "__new__",
    "__call__",
    "__setattr__",
}

BINARY_MAGIC_METHODS: Final = {
    "__add__",
    "__and__",
    "__divmod__",
    "__eq__",
    "__floordiv__",
    "__ge__",
    "__gt__",
    "__iadd__",
    "__iand__",
    "__idiv__",
    "__ifloordiv__",
    "__ilshift__",
    "__imatmul__",
    "__imod__",
    "__imul__",
    "__ior__",
    "__ipow__",
    "__irshift__",
    "__isub__",
    "__itruediv__",
    "__ixor__",
    "__le__",
    "__lshift__",
    "__lt__",
    "__matmul__",
    "__mod__",
    "__mul__",
    "__ne__",
    "__or__",
    "__pow__",
    "__radd__",
    "__rand__",
    "__rdiv__",
    "__rfloordiv__",
    "__rlshift__",
    "__rmatmul__",
    "__rmod__",
    "__rmul__",
    "__ror__",
    "__rpow__",
    "__rrshift__",
    "__rshift__",
    "__rsub__",
    "__rtruediv__",
    "__rxor__",
    "__sub__",
    "__truediv__",
    "__xor__",
}

assert not (_NON_BINARY_MAGIC_METHODS & BINARY_MAGIC_METHODS)

MAGIC_METHODS: Final = _NON_BINARY_MAGIC_METHODS | BINARY_MAGIC_METHODS

MAGIC_METHODS_POS_ARGS_ONLY: Final = MAGIC_METHODS - MAGIC_METHODS_ALLOWING_KWARGS

def special_function_elide_names(name: str) -> bool:
    return name in MAGIC_METHODS_POS_ARGS_ONLY

def argument_elide_name(name: str | None) -> bool:
    return name is not None and name.startswith("__") and not name.endswith("__")


"""Type inference constraint solving"""

from __future__ import annotations

from collections import defaultdict
from typing import Iterable, Sequence
from typing_extensions import TypeAlias as _TypeAlias

from mypy.constraints import SUBTYPE_OF, SUPERTYPE_OF, Constraint, infer_constraints, neg_op
from mypy.expandtype import expand_type
from mypy.graph_utils import prepare_sccs, strongly_connected_components, topsort
from mypy.join import join_types
from mypy.meet import meet_type_list, meet_types
from mypy.subtypes import is_subtype
from mypy.typeops import get_all_type_vars
from mypy.types import (
    AnyType,
    Instance,
    NoneType,
    ParamSpecType,
    ProperType,
    TupleType,
    Type,
    TypeOfAny,
    TypeVarId,
    TypeVarLikeType,
    TypeVarTupleType,
    TypeVarType,
    UninhabitedType,
    UnionType,
    UnpackType,
    get_proper_type,
)
from mypy.typestate import type_state

Bounds: _TypeAlias = "dict[TypeVarId, set[Type]]"
Graph: _TypeAlias = "set[tuple[TypeVarId, TypeVarId]]"
Solutions: _TypeAlias = "dict[TypeVarId, Type | None]"

def solve_constraints(
    original_vars: Sequence[TypeVarLikeType],
    constraints: list[Constraint],
    strict: bool = True,
    allow_polymorphic: bool = False,
    skip_unsatisfied: bool = False,
) -> tuple[list[Type | None], list[TypeVarLikeType]]:
    """Solve type constraints.

    Return the best type(s) for type variables; each type can be None if the value of
    the variable could not be solved.

    If a variable has no constraints, if strict=True then arbitrarily
    pick UninhabitedType as the value of the type variable. If strict=False, pick AnyType.
    If allow_polymorphic=True, then use the full algorithm that can potentially return
    free type variables in solutions (these require special care when applying). Otherwise,
    use a simplified algorithm that just solves each type variable individually if possible.

    The skip_unsatisfied flag matches the same one in applytype.apply_generic_arguments().
    """
    vars = [tv.id for tv in original_vars]
    if not vars:
        return [], []

    originals = {tv.id: tv for tv in original_vars}
    extra_vars: list[TypeVarId] = []
    # Get additional type variables from generic actuals.
    for c in constraints:
        extra_vars.extend([v.id for v in c.extra_tvars if v.id not in vars + extra_vars])
        originals.update({v.id: v for v in c.extra_tvars if v.id not in originals})

    if allow_polymorphic:
        # Constraints inferred from unions require special handling in polymorphic inference.
        constraints = skip_reverse_union_constraints(constraints)

    # Collect a list of constraints for each type variable.
    cmap: dict[TypeVarId, list[Constraint]] = {tv: [] for tv in vars + extra_vars}
    for con in constraints:
        if con.type_var in vars + extra_vars:
            cmap[con.type_var].append(con)

    if allow_polymorphic:
        if constraints:
            solutions, free_vars = solve_with_dependent(
                vars + extra_vars, constraints, vars, originals
            )
        else:
            solutions = {}
            free_vars = []
    else:
        solutions = {}
        free_vars = []
        for tv, cs in cmap.items():
            if not cs:
                continue
            lowers = [c.target for c in cs if c.op == SUPERTYPE_OF]
            uppers = [c.target for c in cs if c.op == SUBTYPE_OF]
            solution = solve_one(lowers, uppers)

            # Do not leak type variables in non-polymorphic solutions.
            if solution is None or not get_vars(
                solution, [tv for tv in extra_vars if tv not in vars]
            ):
                solutions[tv] = solution

    res: list[Type | None] = []
    for v in vars:
        if v in solutions:
            res.append(solutions[v])
        else:
            # No constraints for type variable -- 'UninhabitedType' is the most specific type.
            candidate: Type
            if strict:
                candidate = UninhabitedType()
                candidate.ambiguous = True
            else:
                candidate = AnyType(TypeOfAny.special_form)
            res.append(candidate)

    if not free_vars and not skip_unsatisfied:
        # Most of the validation for solutions is done in applytype.py, but here we can
        # quickly test solutions w.r.t. to upper bounds, and use the latter (if possible),
        # if solutions are actually not valid (due to poor inference context).
        res = pre_validate_solutions(res, original_vars, constraints)

    return res, free_vars

def solve_with_dependent(
    vars: list[TypeVarId],
    constraints: list[Constraint],
    original_vars: list[TypeVarId],
    originals: dict[TypeVarId, TypeVarLikeType],
) -> tuple[Solutions, list[TypeVarLikeType]]:
    """Solve set of constraints that may depend on each other, like T <: List[S].

    The whole algorithm consists of five steps:
      * Propagate via linear constraints and use secondary constraints to get transitive closure
      * Find dependencies between type variables, group them in SCCs, and sort topologically
      * Check that all SCC are intrinsically linear, we can't solve (express) T <: List[T]
      * Variables in leaf SCCs that don't have constant bounds are free (choose one per SCC)
      * Solve constraints iteratively starting from leafs, updating bounds after each step.
    """
    graph, lowers, uppers = transitive_closure(vars, constraints)

    dmap = compute_dependencies(vars, graph, lowers, uppers)
    sccs = list(strongly_connected_components(set(vars), dmap))
    if not all(check_linear(scc, lowers, uppers) for scc in sccs):
        return {}, []
    raw_batches = list(topsort(prepare_sccs(sccs, dmap)))

    free_vars = []
    free_solutions = {}
    for scc in raw_batches[0]:
        # If there are no bounds on this SCC, then the only meaningful solution we can
        # express, is that each variable is equal to a new free variable. For example,
        # if we have T <: S, S <: U, we deduce: T = S = U = <free>.
        if all(not lowers[tv] and not uppers[tv] for tv in scc):
            best_free = choose_free([originals[tv] for tv in scc], original_vars)
            if best_free:
                # TODO: failing to choose may cause leaking type variables,
                # we need to fail gracefully instead.
                free_vars.append(best_free.id)
                free_solutions[best_free.id] = best_free

    # Update lowers/uppers with free vars, so these can now be used
    # as valid solutions.
    for l, u in graph:
        if l in free_vars:
            lowers[u].add(free_solutions[l])
        if u in free_vars:
            uppers[l].add(free_solutions[u])

    # Flatten the SCCs that are independent, we can solve them together,
    # since we don't need to update any targets in between.
    batches = []
    for batch in raw_batches:
        next_bc = []
        for scc in batch:
            next_bc.extend(list(scc))
        batches.append(next_bc)

    solutions: dict[TypeVarId, Type | None] = {}
    for flat_batch in batches:
        res = solve_iteratively(flat_batch, graph, lowers, uppers)
        solutions.update(res)
    return solutions, [free_solutions[tv] for tv in free_vars]

def solve_iteratively(
    batch: list[TypeVarId], graph: Graph, lowers: Bounds, uppers: Bounds
) -> Solutions:
    """Solve transitive closure sequentially, updating upper/lower bounds after each step.

    Transitive closure is represented as a linear graph plus lower/upper bounds for each
    type variable, see transitive_closure() docstring for details.

    We solve for type variables that appear in `batch`. If a bound is not constant (i.e. it
    looks like T :> F[S, ...]), we substitute solutions found so far in the target F[S, ...]
    after solving the batch.

    Importantly, after solving each variable in a batch, we move it from linear graph to
    upper/lower bounds, this way we can guarantee consistency of solutions (see comment below
    for an example when this is important).
    """
    solutions = {}
    s_batch = set(batch)
    while s_batch:
        for tv in sorted(s_batch, key=lambda x: x.raw_id):
            if lowers[tv] or uppers[tv]:
                solvable_tv = tv
                break
        else:
            break
        # Solve each solvable type variable separately.
        s_batch.remove(solvable_tv)
        result = solve_one(lowers[solvable_tv], uppers[solvable_tv])
        solutions[solvable_tv] = result
        if result is None:
            # TODO: support backtracking lower/upper bound choices and order within SCCs.
            # (will require switching this function from iterative to recursive).
            continue

        # Update the (transitive) bounds from graph if there is a solution.
        # This is needed to guarantee solutions will never contradict the initial
        # constraints. For example, consider {T <: S, T <: A, S :> B} with A :> B.
        # If we would not update the uppers/lowers from graph, we would infer T = A, S = B
        # which is not correct.
        for l, u in graph.copy():
            if l == u:
                continue
            if l == solvable_tv:
                lowers[u].add(result)
                graph.remove((l, u))
            if u == solvable_tv:
                uppers[l].add(result)
                graph.remove((l, u))

    # We can update uppers/lowers only once after solving the whole SCC,
    # since uppers/lowers can't depend on type variables in the SCC
    # (and we would reject such SCC as non-linear and therefore not solvable).
    subs = {tv: s for (tv, s) in solutions.items() if s is not None}
    for tv in lowers:
        lowers[tv] = {expand_type(lt, subs) for lt in lowers[tv]}
    for tv in uppers:
        uppers[tv] = {expand_type(ut, subs) for ut in uppers[tv]}
    return solutions

def solve_one(lowers: Iterable[Type], uppers: Iterable[Type]) -> Type | None:
    """Solve constraints by finding by using meets of upper bounds, and joins of lower bounds."""
    bottom: Type | None = None
    top: Type | None = None
    candidate: Type | None = None

    # Filter out previous results of failed inference, they will only spoil the current pass...
    new_uppers = []
    for u in uppers:
        pu = get_proper_type(u)
        if not isinstance(pu, UninhabitedType) or not pu.ambiguous:
            new_uppers.append(u)
    uppers = new_uppers

    # ...unless this is the only information we have, then we just pass it on.
    if not uppers and not lowers:
        candidate = UninhabitedType()
        candidate.ambiguous = True
        return candidate

    # Process each bound separately, and calculate the lower and upper
    # bounds based on constraints. Note that we assume that the constraint
    # targets do not have constraint references.
    for target in lowers:
        if bottom is None:
            bottom = target
        else:
            if type_state.infer_unions:
                # This deviates from the general mypy semantics because
                # recursive types are union-heavy in 95% of cases.
                bottom = UnionType.make_union([bottom, target])
            else:
                bottom = join_types(bottom, target)

    for target in uppers:
        if top is None:
            top = target
        else:
            top = meet_types(top, target)

    p_top = get_proper_type(top)
    p_bottom = get_proper_type(bottom)
    if isinstance(p_top, AnyType) or isinstance(p_bottom, AnyType):
        source_any = top if isinstance(p_top, AnyType) else bottom
        assert isinstance(source_any, ProperType) and isinstance(source_any, AnyType)
        return AnyType(TypeOfAny.from_another_any, source_any=source_any)
    elif bottom is None:
        if top:
            candidate = top
        else:
            # No constraints for type variable
            return None
    elif top is None:
        candidate = bottom
    elif is_subtype(bottom, top):
        candidate = bottom
    else:
        candidate = None
    return candidate

def choose_free(
    scc: list[TypeVarLikeType], original_vars: list[TypeVarId]
) -> TypeVarLikeType | None:
    """Choose the best solution for an SCC containing only type variables.

    This is needed to preserve e.g. the upper bound in a situation like this:
        def dec(f: Callable[[T], S]) -> Callable[[T], S]: ...

        @dec
        def test(x: U) -> U: ...

    where U <: A.
    """

    if len(scc) == 1:
        # Fast path, choice is trivial.
        return scc[0]

    common_upper_bound = meet_type_list([t.upper_bound for t in scc])
    common_upper_bound_p = get_proper_type(common_upper_bound)
    # We include None for when strict-optional is disabled.
    if isinstance(common_upper_bound_p, (UninhabitedType, NoneType)):
        # This will cause to infer Never, which is better than a free TypeVar
        # that has an upper bound Never.
        return None

    values: list[Type] = []
    for tv in scc:
        if isinstance(tv, TypeVarType) and tv.values:
            if values:
                # It is too tricky to support multiple TypeVars with values
                # within the same SCC.
                return None
            values = tv.values.copy()

    if values and not is_trivial_bound(common_upper_bound_p):
        # If there are both values and upper bound present, we give up,
        # since type variables having both are not supported.
        return None

    # For convenience with current type application machinery, we use a stable
    # choice that prefers the original type variables (not polymorphic ones) in SCC.
    best = sorted(scc, key=lambda x: (x.id not in original_vars, x.id.raw_id))[0]
    if isinstance(best, TypeVarType):
        return best.copy_modified(values=values, upper_bound=common_upper_bound)
    if is_trivial_bound(common_upper_bound_p, allow_tuple=True):
        # TODO: support more cases for ParamSpecs/TypeVarTuples
        return best
    return None

def is_trivial_bound(tp: ProperType, allow_tuple: bool = False) -> bool:
    if isinstance(tp, Instance) and tp.type.fullname == "builtins.tuple":
        return allow_tuple and is_trivial_bound(get_proper_type(tp.args[0]))
    return isinstance(tp, Instance) and tp.type.fullname == "builtins.object"

def find_linear(c: Constraint) -> tuple[bool, TypeVarId | None]:
    """Find out if this constraint represent a linear relationship, return target id if yes."""
    if isinstance(c.origin_type_var, TypeVarType):
        if isinstance(c.target, TypeVarType):
            return True, c.target.id
    if isinstance(c.origin_type_var, ParamSpecType):
        if isinstance(c.target, ParamSpecType) and not c.target.prefix.arg_types:
            return True, c.target.id
    if isinstance(c.origin_type_var, TypeVarTupleType):
        target = get_proper_type(c.target)
        if isinstance(target, TupleType) and len(target.items) == 1:
            item = target.items[0]
            if isinstance(item, UnpackType) and isinstance(item.type, TypeVarTupleType):
                return True, item.type.id
    return False, None

def transitive_closure(
    tvars: list[TypeVarId], constraints: list[Constraint]
) -> tuple[Graph, Bounds, Bounds]:
    """Find transitive closure for given constraints on type variables.

    Transitive closure gives maximal set of lower/upper bounds for each type variable,
    such that we cannot deduce any further bounds by chaining other existing bounds.

    The transitive closure is represented by:
      * A set of lower and upper bounds for each type variable, where only constant and
        non-linear terms are included in the bounds.
      * A graph of linear constraints between type variables (represented as a set of pairs)
    Such separation simplifies reasoning, and allows an efficient and simple incremental
    transitive closure algorithm that we use here.

    For example if we have initial constraints [T <: S, S <: U, U <: int], the transitive
    closure is given by:
      * {} <: T <: {int}
      * {} <: S <: {int}
      * {} <: U <: {int}
      * {T <: S, S <: U, T <: U}
    """
    uppers: Bounds = defaultdict(set)
    lowers: Bounds = defaultdict(set)
    graph: Graph = {(tv, tv) for tv in tvars}

    remaining = set(constraints)
    while remaining:
        c = remaining.pop()
        # Note that ParamSpec constraint P <: Q may be considered linear only if Q has no prefix,
        # for cases like P <: Concatenate[T, Q] we should consider this non-linear and put {P} and
        # {T, Q} into separate SCCs. Similarly, Ts <: Tuple[*Us] considered linear, while
        # Ts <: Tuple[*Us, U] is non-linear.
        is_linear, target_id = find_linear(c)
        if is_linear and target_id in tvars:
            assert target_id is not None
            if c.op == SUBTYPE_OF:
                lower, upper = c.type_var, target_id
            else:
                lower, upper = target_id, c.type_var
            if (lower, upper) in graph:
                continue
            graph |= {
                (l, u) for l in tvars for u in tvars if (l, lower) in graph and (upper, u) in graph
            }
            for u in tvars:
                if (upper, u) in graph:
                    lowers[u] |= lowers[lower]
            for l in tvars:
                if (l, lower) in graph:
                    uppers[l] |= uppers[upper]
            for lt in lowers[lower]:
                for ut in uppers[upper]:
                    add_secondary_constraints(remaining, lt, ut)
        elif c.op == SUBTYPE_OF:
            if c.target in uppers[c.type_var]:
                continue
            for l in tvars:
                if (l, c.type_var) in graph:
                    uppers[l].add(c.target)
            for lt in lowers[c.type_var]:
                add_secondary_constraints(remaining, lt, c.target)
        else:
            assert c.op == SUPERTYPE_OF
            if c.target in lowers[c.type_var]:
                continue
            for u in tvars:
                if (c.type_var, u) in graph:
                    lowers[u].add(c.target)
            for ut in uppers[c.type_var]:
                add_secondary_constraints(remaining, c.target, ut)
    return graph, lowers, uppers

def add_secondary_constraints(cs: set[Constraint], lower: Type, upper: Type) -> None:
    """Add secondary constraints inferred between lower and upper (in place)."""
    if isinstance(get_proper_type(upper), UnionType) and isinstance(
        get_proper_type(lower), UnionType
    ):
        # When both types are unions, this can lead to inferring spurious constraints,
        # for example Union[T, int] <: S <: Union[T, int] may infer T <: int.
        # To avoid this, just skip them for now.
        return
    # TODO: what if secondary constraints result in inference against polymorphic actual?
    cs.update(set(infer_constraints(lower, upper, SUBTYPE_OF)))
    cs.update(set(infer_constraints(upper, lower, SUPERTYPE_OF)))

def compute_dependencies(
    tvars: list[TypeVarId], graph: Graph, lowers: Bounds, uppers: Bounds
) -> dict[TypeVarId, list[TypeVarId]]:
    """Compute dependencies between type variables induced by constraints.

    If we have a constraint like T <: List[S], we say that T depends on S, since
    we will need to solve for S first before we can solve for T.
    """
    res = {}
    for tv in tvars:
        deps = set()
        for lt in lowers[tv]:
            deps |= get_vars(lt, tvars)
        for ut in uppers[tv]:
            deps |= get_vars(ut, tvars)
        for other in tvars:
            if other == tv:
                continue
            if (tv, other) in graph or (other, tv) in graph:
                deps.add(other)
        res[tv] = list(deps)
    return res

def check_linear(scc: set[TypeVarId], lowers: Bounds, uppers: Bounds) -> bool:
    """Check there are only linear constraints between type variables in SCC.

    Linear are constraints like T <: S (while T <: F[S] are non-linear).
    """
    for tv in scc:
        if any(get_vars(lt, list(scc)) for lt in lowers[tv]):
            return False
        if any(get_vars(ut, list(scc)) for ut in uppers[tv]):
            return False
    return True

def skip_reverse_union_constraints(cs: list[Constraint]) -> list[Constraint]:
    """Avoid ambiguities for constraints inferred from unions during polymorphic inference.

    Polymorphic inference implicitly relies on assumption that a reverse of a linear constraint
    is a linear constraint. This is however not true in presence of union types, for example
    T :> Union[S, int] vs S <: T. Trying to solve such constraints would be detected ambiguous
    as (T, S) form a non-linear SCC. However, simply removing the linear part results in a valid
    solution T = Union[S, int], S = <free>. A similar scenario is when we get T <: Union[T, int],
    such constraints carry no information, and will equally confuse linearity check.

    TODO: a cleaner solution may be to avoid inferring such constraints in first place, but
    this would require passing around a flag through all infer_constraints() calls.
    """
    reverse_union_cs = set()
    for c in cs:
        p_target = get_proper_type(c.target)
        if isinstance(p_target, UnionType):
            for item in p_target.items:
                if isinstance(item, TypeVarType):
                    if item == c.origin_type_var and c.op == SUBTYPE_OF:
                        reverse_union_cs.add(c)
                        continue
                    # These two forms are semantically identical, but are different from
                    # the point of view of Constraint.__eq__().
                    reverse_union_cs.add(Constraint(item, neg_op(c.op), c.origin_type_var))
                    reverse_union_cs.add(Constraint(c.origin_type_var, c.op, item))
    return [c for c in cs if c not in reverse_union_cs]

def get_vars(target: Type, vars: list[TypeVarId]) -> set[TypeVarId]:
    """Find type variables for which we are solving in a target type."""
    return {tv.id for tv in get_all_type_vars(target)} & set(vars)

def pre_validate_solutions(
    solutions: list[Type | None],
    original_vars: Sequence[TypeVarLikeType],
    constraints: list[Constraint],
) -> list[Type | None]:
    """Check is each solution satisfies the upper bound of the corresponding type variable.

    If it doesn't satisfy the bound, check if bound itself satisfies all constraints, and
    if yes, use it instead as a fallback solution.
    """
    new_solutions: list[Type | None] = []
    for t, s in zip(original_vars, solutions):
        if is_callable_protocol(t.upper_bound):
            # This is really ad-hoc, but a proper fix would be much more complex,
            # and otherwise this may cause crash in a relatively common scenario.
            new_solutions.append(s)
            continue
        if s is not None and not is_subtype(s, t.upper_bound):
            bound_satisfies_all = True
            for c in constraints:
                if c.op == SUBTYPE_OF and not is_subtype(t.upper_bound, c.target):
                    bound_satisfies_all = False
                    break
                if c.op == SUPERTYPE_OF and not is_subtype(c.target, t.upper_bound):
                    bound_satisfies_all = False
                    break
            if bound_satisfies_all:
                new_solutions.append(t.upper_bound)
                continue
        new_solutions.append(s)
    return new_solutions

def is_callable_protocol(t: Type) -> bool:
    proper_t = get_proper_type(t)
    if isinstance(proper_t, Instance) and proper_t.type.is_protocol:
        return "__call__" in proper_t.type.protocol_members
    return False


"""Split namespace for argparse to allow separating options by prefix.

We use this to direct some options to an Options object and some to a
regular namespace.
"""

# In its own file largely because mypyc doesn't support its use of
# __getattr__/__setattr__ and has some issues with __dict__

from __future__ import annotations

import argparse
from typing import Any

class SplitNamespace(argparse.Namespace):
    def __init__(self, standard_namespace: object, alt_namespace: object, alt_prefix: str) -> None:
        self.__dict__["_standard_namespace"] = standard_namespace
        self.__dict__["_alt_namespace"] = alt_namespace
        self.__dict__["_alt_prefix"] = alt_prefix

    def _get(self) -> tuple[Any, Any]:
        return (self._standard_namespace, self._alt_namespace)

    def __setattr__(self, name: str, value: Any) -> None:
        if name.startswith(self._alt_prefix):
            setattr(self._alt_namespace, name[len(self._alt_prefix) :], value)
        else:
            setattr(self._standard_namespace, name, value)

    def __getattr__(self, name: str) -> Any:
        if name.startswith(self._alt_prefix):
            return getattr(self._alt_namespace, name[len(self._alt_prefix) :])
        else:
            return getattr(self._standard_namespace, name)


from __future__ import annotations

from contextlib import contextmanager
from typing import Final, Iterator

# These are global mutable state. Don't add anything here unless there's a very
# good reason.

class StrictOptionalState:
    # Wrap this in a class since it's faster that using a module-level attribute.

    def __init__(self, strict_optional: bool) -> None:
        # Value varies by file being processed
        self.strict_optional = strict_optional

    @contextmanager
    def strict_optional_set(self, value: bool) -> Iterator[None]:
        saved = self.strict_optional
        self.strict_optional = value
        try:
            yield
        finally:
            self.strict_optional = saved

state: Final = StrictOptionalState(strict_optional=False)
find_occurrences: tuple[str, str] | None = None


"""Utilities for calculating and reporting statistics about types."""

from __future__ import annotations

import os
from collections import Counter
from contextlib import contextmanager
from typing import Final, Iterator

from mypy import nodes
from mypy.argmap import map_formals_to_actuals
from mypy.nodes import (
    AssignmentExpr,
    AssignmentStmt,
    BreakStmt,
    BytesExpr,
    CallExpr,
    ClassDef,
    ComparisonExpr,
    ComplexExpr,
    ContinueStmt,
    EllipsisExpr,
    Expression,
    ExpressionStmt,
    FloatExpr,
    FuncDef,
    Import,
    ImportAll,
    ImportFrom,
    IndexExpr,
    IntExpr,
    MemberExpr,
    MypyFile,
    NameExpr,
    Node,
    OpExpr,
    PassStmt,
    RefExpr,
    StrExpr,
    TypeApplication,
    UnaryExpr,
    YieldFromExpr,
)
from mypy.traverser import TraverserVisitor
from mypy.typeanal import collect_all_inner_types
from mypy.types import (
    AnyType,
    CallableType,
    FunctionLike,
    Instance,
    TupleType,
    Type,
    TypeOfAny,
    TypeQuery,
    TypeVarType,
    get_proper_type,
    get_proper_types,
)
from mypy.util import correct_relative_import

TYPE_EMPTY: Final = 0
TYPE_UNANALYZED: Final = 1  # type of non-typechecked code
TYPE_PRECISE: Final = 2
TYPE_IMPRECISE: Final = 3
TYPE_ANY: Final = 4

precision_names: Final = ["empty", "unanalyzed", "precise", "imprecise", "any"]

class StatisticsVisitor(TraverserVisitor):
    def __init__(
        self,
        inferred: bool,
        filename: str,
        modules: dict[str, MypyFile],
        typemap: dict[Expression, Type] | None = None,
        all_nodes: bool = False,
        visit_untyped_defs: bool = True,
    ) -> None:
        self.inferred = inferred
        self.filename = filename
        self.modules = modules
        self.typemap = typemap
        self.all_nodes = all_nodes
        self.visit_untyped_defs = visit_untyped_defs

        self.num_precise_exprs = 0
        self.num_imprecise_exprs = 0
        self.num_any_exprs = 0

        self.num_simple_types = 0
        self.num_generic_types = 0
        self.num_tuple_types = 0
        self.num_function_types = 0
        self.num_typevar_types = 0
        self.num_complex_types = 0
        self.num_any_types = 0

        self.line = -1

        self.line_map: dict[int, int] = {}

        self.type_of_any_counter: Counter[int] = Counter()
        self.any_line_map: dict[int, list[AnyType]] = {}

        # For each scope (top level/function), whether the scope was type checked
        # (annotated function).
        #
        # TODO: Handle --check-untyped-defs
        self.checked_scopes = [True]

        self.output: list[str] = []

        TraverserVisitor.__init__(self)

    def visit_mypy_file(self, o: MypyFile) -> None:
        self.cur_mod_node = o
        self.cur_mod_id = o.fullname
        super().visit_mypy_file(o)

    def visit_import_from(self, imp: ImportFrom) -> None:
        self.process_import(imp)

    def visit_import_all(self, imp: ImportAll) -> None:
        self.process_import(imp)

    def process_import(self, imp: ImportFrom | ImportAll) -> None:
        import_id, ok = correct_relative_import(
            self.cur_mod_id, imp.relative, imp.id, self.cur_mod_node.is_package_init_file()
        )
        if ok and import_id in self.modules:
            kind = TYPE_PRECISE
        else:
            kind = TYPE_ANY
        self.record_line(imp.line, kind)

    def visit_import(self, imp: Import) -> None:
        if all(id in self.modules for id, _ in imp.ids):
            kind = TYPE_PRECISE
        else:
            kind = TYPE_ANY
        self.record_line(imp.line, kind)

    def visit_func_def(self, o: FuncDef) -> None:
        with self.enter_scope(o):
            self.line = o.line
            if len(o.expanded) > 1 and o.expanded != [o] * len(o.expanded):
                if o in o.expanded:
                    print(
                        "{}:{}: ERROR: cycle in function expansion; skipping".format(
                            self.filename, o.line
                        )
                    )
                    return
                for defn in o.expanded:
                    assert isinstance(defn, FuncDef)
                    self.visit_func_def(defn)
            else:
                if o.type:
                    assert isinstance(o.type, CallableType)
                    sig = o.type
                    arg_types = sig.arg_types
                    if sig.arg_names and sig.arg_names[0] == "self" and not self.inferred:
                        arg_types = arg_types[1:]
                    for arg in arg_types:
                        self.type(arg)
                    self.type(sig.ret_type)
                elif self.all_nodes:
                    self.record_line(self.line, TYPE_ANY)
                if not o.is_dynamic() or self.visit_untyped_defs:
                    super().visit_func_def(o)

    @contextmanager
    def enter_scope(self, o: FuncDef) -> Iterator[None]:
        self.checked_scopes.append(o.type is not None and self.checked_scopes[-1])
        yield None
        self.checked_scopes.pop()

    def is_checked_scope(self) -> bool:
        return self.checked_scopes[-1]

    def visit_class_def(self, o: ClassDef) -> None:
        self.record_line(o.line, TYPE_PRECISE)  # TODO: Look at base classes
        # Override this method because we don't want to analyze base_type_exprs (base_type_exprs
        # are base classes in a class declaration).
        # While base_type_exprs are technically expressions, type analyzer does not visit them and
        # they are not in the typemap.
        for d in o.decorators:
            d.accept(self)
        o.defs.accept(self)

    def visit_type_application(self, o: TypeApplication) -> None:
        self.line = o.line
        for t in o.types:
            self.type(t)
        super().visit_type_application(o)

    def visit_assignment_stmt(self, o: AssignmentStmt) -> None:
        self.line = o.line
        if isinstance(o.rvalue, nodes.CallExpr) and isinstance(
            o.rvalue.analyzed, nodes.TypeVarExpr
        ):
            # Type variable definition -- not a real assignment.
            return
        if o.type:
            # If there is an explicit type, don't visit the l.h.s. as an expression
            # to avoid double-counting and mishandling special forms.
            self.type(o.type)
            o.rvalue.accept(self)
            return
        elif self.inferred and not self.all_nodes:
            # if self.all_nodes is set, lvalues will be visited later
            for lvalue in o.lvalues:
                if isinstance(lvalue, nodes.TupleExpr):
                    items = lvalue.items
                else:
                    items = [lvalue]
                for item in items:
                    if isinstance(item, RefExpr) and item.is_inferred_def:
                        if self.typemap is not None:
                            self.type(self.typemap.get(item))
        super().visit_assignment_stmt(o)

    def visit_expression_stmt(self, o: ExpressionStmt) -> None:
        if isinstance(o.expr, (StrExpr, BytesExpr)):
            # Docstring
            self.record_line(o.line, TYPE_EMPTY)
        else:
            super().visit_expression_stmt(o)

    def visit_pass_stmt(self, o: PassStmt) -> None:
        self.record_precise_if_checked_scope(o)

    def visit_break_stmt(self, o: BreakStmt) -> None:
        self.record_precise_if_checked_scope(o)

    def visit_continue_stmt(self, o: ContinueStmt) -> None:
        self.record_precise_if_checked_scope(o)

    def visit_name_expr(self, o: NameExpr) -> None:
        if o.fullname in ("builtins.None", "builtins.True", "builtins.False", "builtins.Ellipsis"):
            self.record_precise_if_checked_scope(o)
        else:
            self.process_node(o)
            super().visit_name_expr(o)

    def visit_yield_from_expr(self, o: YieldFromExpr) -> None:
        if o.expr:
            o.expr.accept(self)

    def visit_call_expr(self, o: CallExpr) -> None:
        self.process_node(o)
        if o.analyzed:
            o.analyzed.accept(self)
        else:
            o.callee.accept(self)
            for a in o.args:
                a.accept(self)
            self.record_call_target_precision(o)

    def record_call_target_precision(self, o: CallExpr) -> None:
        """Record precision of formal argument types used in a call."""
        if not self.typemap or o.callee not in self.typemap:
            # Type not available.
            return
        callee_type = get_proper_type(self.typemap[o.callee])
        if isinstance(callee_type, CallableType):
            self.record_callable_target_precision(o, callee_type)
        else:
            pass  # TODO: Handle overloaded functions, etc.

    def record_callable_target_precision(self, o: CallExpr, callee: CallableType) -> None:
        """Record imprecision caused by callee argument types.

        This only considers arguments passed in a call expression. Arguments
        with default values that aren't provided in a call arguably don't
        contribute to typing imprecision at the *call site* (but they
        contribute at the function definition).
        """
        assert self.typemap
        typemap = self.typemap
        actual_to_formal = map_formals_to_actuals(
            o.arg_kinds,
            o.arg_names,
            callee.arg_kinds,
            callee.arg_names,
            lambda n: typemap[o.args[n]],
        )
        for formals in actual_to_formal:
            for n in formals:
                formal = get_proper_type(callee.arg_types[n])
                if isinstance(formal, AnyType):
                    self.record_line(o.line, TYPE_ANY)
                elif is_imprecise(formal):
                    self.record_line(o.line, TYPE_IMPRECISE)

    def visit_member_expr(self, o: MemberExpr) -> None:
        self.process_node(o)
        super().visit_member_expr(o)

    def visit_op_expr(self, o: OpExpr) -> None:
        self.process_node(o)
        super().visit_op_expr(o)

    def visit_comparison_expr(self, o: ComparisonExpr) -> None:
        self.process_node(o)
        super().visit_comparison_expr(o)

    def visit_index_expr(self, o: IndexExpr) -> None:
        self.process_node(o)
        super().visit_index_expr(o)

    def visit_assignment_expr(self, o: AssignmentExpr) -> None:
        self.process_node(o)
        super().visit_assignment_expr(o)

    def visit_unary_expr(self, o: UnaryExpr) -> None:
        self.process_node(o)
        super().visit_unary_expr(o)

    def visit_str_expr(self, o: StrExpr) -> None:
        self.record_precise_if_checked_scope(o)

    def visit_bytes_expr(self, o: BytesExpr) -> None:
        self.record_precise_if_checked_scope(o)

    def visit_int_expr(self, o: IntExpr) -> None:
        self.record_precise_if_checked_scope(o)

    def visit_float_expr(self, o: FloatExpr) -> None:
        self.record_precise_if_checked_scope(o)

    def visit_complex_expr(self, o: ComplexExpr) -> None:
        self.record_precise_if_checked_scope(o)

    def visit_ellipsis(self, o: EllipsisExpr) -> None:
        self.record_precise_if_checked_scope(o)

    # Helpers

    def process_node(self, node: Expression) -> None:
        if self.all_nodes:
            if self.typemap is not None:
                self.line = node.line
                self.type(self.typemap.get(node))

    def record_precise_if_checked_scope(self, node: Node) -> None:
        if isinstance(node, Expression) and self.typemap and node not in self.typemap:
            kind = TYPE_UNANALYZED
        elif self.is_checked_scope():
            kind = TYPE_PRECISE
        else:
            kind = TYPE_ANY
        self.record_line(node.line, kind)

    def type(self, t: Type | None) -> None:
        t = get_proper_type(t)

        if not t:
            # If an expression does not have a type, it is often due to dead code.
            # Don't count these because there can be an unanalyzed value on a line with other
            # analyzed expressions, which overwrite the TYPE_UNANALYZED.
            self.record_line(self.line, TYPE_UNANALYZED)
            return

        if isinstance(t, AnyType) and is_special_form_any(t):
            # TODO: What if there is an error in special form definition?
            self.record_line(self.line, TYPE_PRECISE)
            return

        if isinstance(t, AnyType):
            self.log("  !! Any type around line %d" % self.line)
            self.num_any_exprs += 1
            self.record_line(self.line, TYPE_ANY)
        elif (not self.all_nodes and is_imprecise(t)) or (self.all_nodes and is_imprecise2(t)):
            self.log("  !! Imprecise type around line %d" % self.line)
            self.num_imprecise_exprs += 1
            self.record_line(self.line, TYPE_IMPRECISE)
        else:
            self.num_precise_exprs += 1
            self.record_line(self.line, TYPE_PRECISE)

        for typ in get_proper_types(collect_all_inner_types(t)) + [t]:
            if isinstance(typ, AnyType):
                typ = get_original_any(typ)
                if is_special_form_any(typ):
                    continue
                self.type_of_any_counter[typ.type_of_any] += 1
                self.num_any_types += 1
                if self.line in self.any_line_map:
                    self.any_line_map[self.line].append(typ)
                else:
                    self.any_line_map[self.line] = [typ]
            elif isinstance(typ, Instance):
                if typ.args:
                    if any(is_complex(arg) for arg in typ.args):
                        self.num_complex_types += 1
                    else:
                        self.num_generic_types += 1
                else:
                    self.num_simple_types += 1
            elif isinstance(typ, FunctionLike):
                self.num_function_types += 1
            elif isinstance(typ, TupleType):
                if any(is_complex(item) for item in typ.items):
                    self.num_complex_types += 1
                else:
                    self.num_tuple_types += 1
            elif isinstance(typ, TypeVarType):
                self.num_typevar_types += 1

    def log(self, string: str) -> None:
        self.output.append(string)

    def record_line(self, line: int, precision: int) -> None:
        self.line_map[line] = max(precision, self.line_map.get(line, TYPE_EMPTY))

def dump_type_stats(
    tree: MypyFile,
    path: str,
    modules: dict[str, MypyFile],
    inferred: bool = False,
    typemap: dict[Expression, Type] | None = None,
) -> None:
    if is_special_module(path):
        return
    print(path)
    visitor = StatisticsVisitor(inferred, filename=tree.fullname, modules=modules, typemap=typemap)
    tree.accept(visitor)
    for line in visitor.output:
        print(line)
    print("  ** precision **")
    print("  precise  ", visitor.num_precise_exprs)
    print("  imprecise", visitor.num_imprecise_exprs)
    print("  any      ", visitor.num_any_exprs)
    print("  ** kinds **")
    print("  simple   ", visitor.num_simple_types)
    print("  generic  ", visitor.num_generic_types)
    print("  function ", visitor.num_function_types)
    print("  tuple    ", visitor.num_tuple_types)
    print("  TypeVar  ", visitor.num_typevar_types)
    print("  complex  ", visitor.num_complex_types)
    print("  any      ", visitor.num_any_types)

def is_special_module(path: str) -> bool:
    return os.path.basename(path) in ("abc.pyi", "typing.pyi", "builtins.pyi")

def is_imprecise(t: Type) -> bool:
    return t.accept(HasAnyQuery())

class HasAnyQuery(TypeQuery[bool]):
    def __init__(self) -> None:
        super().__init__(any)

    def visit_any(self, t: AnyType) -> bool:
        return not is_special_form_any(t)

def is_imprecise2(t: Type) -> bool:
    return t.accept(HasAnyQuery2())

class HasAnyQuery2(HasAnyQuery):
    def visit_callable_type(self, t: CallableType) -> bool:
        # We don't want to flag references to functions with some Any
        # argument types (etc.) since they generally don't mean trouble.
        return False

def is_generic(t: Type) -> bool:
    t = get_proper_type(t)
    return isinstance(t, Instance) and bool(t.args)

def is_complex(t: Type) -> bool:
    t = get_proper_type(t)
    return is_generic(t) or isinstance(t, (FunctionLike, TupleType, TypeVarType))

def is_special_form_any(t: AnyType) -> bool:
    return get_original_any(t).type_of_any == TypeOfAny.special_form

def get_original_any(t: AnyType) -> AnyType:
    if t.type_of_any == TypeOfAny.from_another_any:
        assert t.source_any
        assert t.source_any.type_of_any != TypeOfAny.from_another_any
        t = t.source_any
    return t