Based on the OCaml rewatch reimplementation from #8653, we can prototype integrating the compiler directly into the rewatch executable to avoid having to spawn a bsc binary for each compilation step.
@cristianoc fed the prompt below into Fable, and I fed the same one into Astra.
I'll post the responses we got as separate comments. These can serve as a basis for further discussion.
We are porting Rewatch to OCaml as a drop-in for the Rust version. Next I want to see whether in-process bsc is worth it: stop spawning a bsc process per module, and ideally stop writing cmj/cmt (and maybe cmi) to disk at all.
Do not implement yet. Think this through with me.
Context:
- A lot of build time is process creation and artifact I/O, not typechecking.
- Integration might be >2x. I want a dirty prototype that measures that, even if it is incorrect and single-threaded.
- The compiler has lots of global/shared mutable state (Clflags, Config.load_path, Env cache, Ident, Js_config, typechecker mutation, etc.). There are essentially no tests that reuse one compiler process across modules.
- Getting rid of global state is probably much easier than the Rewatch port, but only if we know what to reset/isolate.
- No multicore required for the prototype. Sequential in-process with no cm* writes may already be faster.
- A real compiler service / parallel-from-one-process comes later.
Please:
- Survey how bsc is invoked today (parse vs compile), what it reads/writes (ast, cmi, cmj, cmt, js), and which globals would leak across compilations.
- Propose the smallest prototype that can give a real speedup signal on stdlib + tests (or a real project). What can stay on disk? What must be in memory? What must be reset between modules?
- Argue whether that prototype is the right next step, or whether something else is better (reset-and-reuse API, keep writing cmi only, compiler service still spawning-free but still on disk, etc.).
- Explicitly list what is likely to go well, what is likely to go wrong, and how we would get fooled by a misleading benchmark.
- Recommend a concrete goal we could later implement with /goal.
Based on the OCaml rewatch reimplementation from #8653, we can prototype integrating the compiler directly into the rewatch executable to avoid having to spawn a bsc binary for each compilation step.
@cristianoc fed the prompt below into Fable, and I fed the same one into Astra.
I'll post the responses we got as separate comments. These can serve as a basis for further discussion.