Skip to content

Presentation compiler re-parses and re-typechecks all shared units on every request #191

Open

Description

@sebaciv

Problem

Shared sources compiled for a profile, such as expression utils, profile objects and Java getter adapters, are kept loaded in the presentation compiler (ScexPresentationCompiler.compile calls askLoadedTyped(sourceFile, keepLoaded = true, …)). Because of how scala.tools.nsc.interactive.Global works, every completion or getErrors request makes all of them stale. The background compiler then re-parses, re-enters and re-typechecks every shared unit, even though they never change after they are compiled.

How a single request causes it

  1. askLoadedTyped(expressionSource) loads a new source, which calls reloadSources and then newTyperRun() and minRunId = currentRunId. Every loaded shared unit is now out of date.
  2. The expression is typechecked, and outOfDate triggers a backgroundCompile pass over all loaded units. That pass runs reset(unit) (clearing synthetics, contexts, …), then parseAndEnter, then typeCheck.
  3. askFilesDeleted(expressionSource) at the end of the request does not start a new run, but it calls demandNewCompilerRun(). If a background pass is running, that throws FreshRunReq and aborts it partway through, often while a shared unit is half-typechecked.
  4. The next request repeats steps 1–3.

Why it matters

  • Wasted work: every request re-typechecks every shared unit of every profile used so far. The cost grows with the number of profiles and with adapter or utils size.
  • Races: it keeps creating half-reset and half-typechecked states, and scex's multi-step requests can see them. Two known bugs come from this:
  • The reports that disabling getter adapters or caches "fixes" these bugs are a side effect of this churn. Those settings change how many shared units are loaded, and therefore how long each pass takes.

Constraint

The presentation compiler currently has no other way to see shared symbols:

  • It is created with plain settings (presentationCompilerSettings). ClassfileReusingScexCompiler only adds the classfile directory to the regular compiler's settings (compilerSettings).
  • Without -SCEXclassfile-directory, shared classes are written to an in-memory VirtualDirectory (sharedClassLoader) that is not on any classpath.

So simply not keeping shared sources loaded would break completion.

Possible directions

  1. Put shared classfiles on the presentation compiler's classpath and stop loading shared sources. This is the only option that removes the churn completely.
    • Needs a classpath entry for the in-memory output, e.g. VirtualDirectoryClassPath.
    • The presentation compiler caches package contents, so newly compiled shared classes must become visible: invalidate the classpath entry or rescan the package.
    • Annotations that completion relies on (@Documentation, @ParameterNames) must survive the round-trip through pickles.
  2. Reduce aborts without changing what is loaded. For example, delete expression sources lazily or in batches, or reuse a single expression source. This removes the extra FreshRunReq from askFilesDeleted, but every new expression still needs reloadSources, which starts a new run. Shared units would still be re-typechecked on every request, so this only shortens the race windows.
  3. Make each scex request one step from the presentation compiler's point of view. Do typecheck, scope/type completion and error collection inside a single work item, so a background pass can't run in between. This addresses the races, including Completer.getErrors gives quite a lot of false positives #17, but not the wasted work.

Option 1 looks like the real fix. Option 3 is a complementary hardening step.

🤖 Written by Claude.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions