Skip to content

feat(repl): analyze each loaded file as a workspace document of its own - #309

Open
devin-ai-integration[bot] wants to merge 32 commits into
developfrom
feature/per-file-documents
Open

devin-ai-integration[bot] wants to merge 32 commits into
developfrom
feature/per-file-documents

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

What and why

Files loaded from the command line (-validate, -satisfy, -check, -run, …) or by %load were joined into the transcript document (<repl>, with a synthetic <repl>.kerml for KerML files) and analyzed as one text. That buffer had semantics of its own: a root-level import in one file served every other file, and two files declaring the same root package were reported as a duplicate. The editor, the corpus gates and the pilot tools analyze the same files through model.Workspace, one document per file, and report the opposite — unresolved reference: Real in the file that does not import it, and no duplicate.

Each loaded file is now a workspace document under its own name, indexed together with the others and analyzed on its own; typed submissions still form the one joined transcript document. Concretely, in internal/repl:

Session.openDocuments()      opens <repl> for the typed text (loaded files masked out) and one
                             document per loaded file, and removes documents of dropped snippets
Session.symbolIndex()        adds every session document to one index, expands wildcards once,
                             takes back documents that went (idxDocs)
Session.locatedDocs()        every session document with its base offset in the joined buffer
Session.sessionMembers()     []Member{Node, Offset, scope}: top-level members of every document,
                             in buffer order, offsets translated to the joined buffer
Session.diagnostics()        transcript + per-file diagnostics, offsets translated, sorted
Result.Members               []ast.Node -> []Member (loaded documents have offsets of their own)

Readers of s.ws.Document(docName) that assumed every declaration lived in <repl>%calc, %sweep, %analyze, %print, the compound-expression path of %eval, promptScope, symbolsInLoadOrder (%view) — now go through hasDeclarations() / sessionDocs() / sessionMembers() / rootScopeOf(sym); nothing re-joins the files behind the scenes. %print can print a declaration of any loaded file rather than only of <repl>/<repl>.kerml. %save is unchanged: it writes each snippet's own text back.

Two user-visible changes for multi-file loads, both the behavior model.Workspace already had:

  • A root-level import in one file no longer serves the other files, nor — after %load a.sysml — the text typed at the prompt. Root packages remain reachable through the global namespace as before, and the prompt still evaluates in the last namespace declared, which may be one a loaded file declares.
  • Two files declaring package A are two root namespaces of that name, not a duplicate; a reference to A resolves to the declaration in the file whose name sorts first — the order the workspace gives documents, whatever order the files were given in on the command line (TestRepeatedRootPackageResolvesByDocumentNameNotLoadOrder pins both orders).

The compound-expression path of %eval parses the expression appended to the transcript with the loaded files masked out (as the transcript document has them), so a loaded file's text can no longer fail the parse of a prompt expression; the expression is still evaluated in promptScope().

This is the first of two PRs: the parallel batch pipeline over these documents (parallel parse, one index and one wildcard expansion per batch, a worker pool with a private passes.Context per worker, diagnostics in document order, a workers setting, the -split-planes generator mode and the measurements) follows in #312. Merge order: this PR first, then #312.

Existing internal/repl tests were examined individually; none asserted the import leak or the duplicate, so none was changed. TestTargetlessPseudoViewSpansLoadedDocuments and TestLoadFilesSummaryMatchesLoadingEachFile failed during the migration and were fixed in the code (symbolsInLoadOrder sorting by buffer offset; Result.ownMembers filtering by the buffer offset), not in the tests.

Known behaviour

Two prompt evaluation rules predate this change and are kept as they are (documented under "Known behaviour" in the loading section of docs/guide/04-repl.md); both are open to change. A prompt expression evaluates in the last namespace declared (promptScope), so when a loaded file declares it, the expression sees that file's root imports although a typed declaration does not — %eval 1.5 as Real resolves after loading private import ScalarValues::*; package A { … } while a typed attribute y : Real; reports Real unresolved; the alternative is a fallback to the transcript's own root. A qualified command argument (%eval A::y, %print A::y) goes through the single index-based lookupSymbol, which holds every document's declarations, so with two loaded package A it reaches the second A's member that a model reference or a compound expression cannot; the alternatives are a resolver-based lookup for the evaluating commands, or rejecting a root that resolves ambiguously. Both were already so on develop (the second for a .kerml/.sysml pair); per-file loading makes them reachable with any two files.

Specification basis

KerML 8.2.3.3 and 8.2.4: a root-level import surfaces its names in the importing document's own root namespace (docs/project/spec-compliance.md, row "A root-level import X::* surfaces its names in the importing document's own root namespace"). KerML 8.2.3.5: two root namespaces of the same name are neither an ambiguity nor a duplicate, and resolution uses the first declaration (docs/project/pilot-xpect.md, "The global namespace…", item 1; TestNameResolutionPassResolvesARepeatedTopLevelNameToTheFirst). No compliance row moves; the command line converges on what those rows already certify for the workspace.

How it was verified

New in internal/repl/filedocs_test.go:

  • TestLoadedFilesDoNotShareRootImportsa.sysml with private import ScalarValues::*;, b.sysml using Real: the command line reports unresolved reference: Real in b.sysml, and its diagnostics equal model.Workspace's over the same files.
  • TestLoadedFilesDeclaringOneRootPackageAreNotDuplicates — two package A files plus a file referencing A::X (first declaration) and A::Y (second): no duplicate, A::X resolves, diagnostics equal the workspace's.
  • TestPromptDoesNotSeeALoadedFilesRootImports — after %load a.sysml, typed text still reaches A but not a bare Real.
  • TestCommandLineLoadMatchesWorkspace — every multi-file directory under testdata/, examples/ and the four OMG corpus roots (examples/sysml-v2-training, examples/pilot-corpora/{kerml-examples,sysml-examples,sysml-validation}) run through the command-line load and through model.Workspace yields the same diagnostics. All four fail on develop.

Gates, all run locally:

gofmt -l .                       (nothing)
go build ./...                   ok
go vet ./...                     ok
go test ./...                    ok
go test -race ./...              ok
OPENSYSML_REQUIRE_TRAINING_CORPUS=1 OPENSYSML_REQUIRE_PILOT_CORPORA=1 \
  go test -count=1 ./internal/core/model -run 'TestTrainingExamples|TestPilotCorpora'   ok
make docs-check                  ok
make docs-counts                 (test-function count moved by the four new tests; README.md and
                                 docs/project/spec-compliance.md regenerated)

training_examples_expected.txt is unchanged and empty; no pilot ratchet moved.

By hand: sysml -validate a.sysml b.sysml on the import case (unresolved reference: Real in b.sysml, exit 2) and on the same-name-package case (clean; the existing note that A is opened by more than one loaded file still prints), and %load of several files followed by %print, %view, %eval of a compound expression and %save (each file's text written back verbatim). The REPL's own name lookup (%print A, %view A) still reports a root name two loaded files declare as ambiguous, as it did before; model resolution (A::X) takes the first declaration.

Checklist

  • make test and make lint pass locally
  • Tests added or updated for the change
  • Documentation extended where it already covers the surface (docs/reference/cli.md "Multiple Files", docs/guide/04-repl.md loading section)
  • Changelog entry added as changes/unreleased/per-file-documents.changed.md, not as an edit to CHANGELOG.md
  • baselines regenerated and make docs-counts run if a gate count moved
  • No internal work-item labels in the body, docs, or changelog

devin-ai-integration Bot and others added 2 commits September 15, 2026 06:36
Files loaded from the command line or by %load were joined into the
transcript document, so a root-level import in one file served the others
and two files declaring one root package were reported as duplicates. Each
loaded file is now a workspace document under its own name, indexed with
the others and analyzed on its own, as the editor and the corpus gates
analyze it; the typed transcript stays one joined document. A differential
test runs every multi-file directory of the fixtures and the OMG corpora
through the command line and a workspace and asserts the same diagnostics.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
…pt alone

Co-Authored-By: jason.han <hanhuijun@gmail.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

…ing skill

Co-Authored-By: jason.han <hanhuijun@gmail.com>
@devin-ai-integration
devin-ai-integration Bot marked this pull request as ready for review September 15, 2026 07:31
devin-ai-integration[bot]

This comment was marked as resolved.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

A loaded file is analyzed as a document of its own, so an error in it gates
that file's deeper checks only. The blocker note on a clean prompt submission
now skips diagnostics from loaded files, and a load's from the transcript.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

A load shares no document with the rest of the buffer, so nothing blocks it and
it neither names nor forgets the error the transcript has already been told of.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration Bot and others added 2 commits September 15, 2026 14:18
… interval

A load still names no blocker, but when it leaves the transcript unblocked the
recorded note is cleared, so the error is named again should a reload bring it back.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
…ocuments

Co-Authored-By: jason.han <hanhuijun@gmail.com>

# Conflicts:
#	README.md
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration Bot and others added 3 commits September 15, 2026 14:57
… document is left

A file reloaded with its enclosure left open is masked and its workspace
document removed; with no scoped document left, symbolIndex returned before
taking the file's previous declarations back out of the session index, so a
qualified lookup kept answering with what the session no longer held. The
empty-document path now drops every indexed document, as a reset does, and
keeps the standard library.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
…ocuments

Co-Authored-By: jason.han <hanhuijun@gmail.com>

# Conflicts:
#	README.md
#	docs/project/spec-compliance.md
…ocuments

Co-Authored-By: jason.han <hanhuijun@gmail.com>

# Conflicts:
#	README.md
#	docs/project/spec-compliance.md
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration Bot and others added 6 commits September 15, 2026 18:19
Co-Authored-By: jason.han <hanhuijun@gmail.com>
… load order

Two files declaring the same root package are ordered as the workspace
orders documents, by name, so which declaration a reference reaches does
not depend on the order the files were given in. A test pins both orders.

Co-Authored-By: jason.han <hanhuijun@gmail.com>
Co-Authored-By: jason.han <hanhuijun@gmail.com>
…er-file loading

Co-Authored-By: jason.han <hanhuijun@gmail.com>
…ocuments

Co-Authored-By: jason.han <hanhuijun@gmail.com>

# Conflicts:
#	README.md
Co-Authored-By: jason.han <hanhuijun@gmail.com>
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration Bot and others added 14 commits September 15, 2026 22:41
…ocuments

Co-Authored-By: jason.han <hanhuijun@gmail.com>

# Conflicts:
#	README.md
#	docs/project/spec-compliance.md
Co-Authored-By: jason.han <hanhuijun@gmail.com>
…ocuments

Co-Authored-By: jason.han <hanhuijun@gmail.com>

# Conflicts:
#	README.md
#	docs/project/spec-compliance.md
Co-Authored-By: jason.han <hanhuijun@gmail.com>
…ocuments

Co-Authored-By: jason.han <hanhuijun@gmail.com>

# Conflicts:
#	README.md
#	docs/project/spec-compliance.md
…ocuments

Co-Authored-By: jason.han <hanhuijun@gmail.com>

# Conflicts:
#	README.md
#	docs/project/spec-compliance.md
Co-Authored-By: jason.han <hanhuijun@gmail.com>
…ocuments

Co-Authored-By: jason.han <hanhuijun@gmail.com>

# Conflicts:
#	README.md
#	docs/project/spec-compliance.md
…ocuments

Co-Authored-By: jason.han <hanhuijun@gmail.com>

# Conflicts:
#	README.md
#	docs/project/spec-compliance.md
…ocuments

Co-Authored-By: jason.han <hanhuijun@gmail.com>

# Conflicts:
#	README.md
#	docs/project/spec-compliance.md
…ocuments

Co-Authored-By: jason.han <hanhuijun@gmail.com>

# Conflicts:
#	README.md
#	docs/project/spec-compliance.md
…ocuments

Co-Authored-By: jason.han <hanhuijun@gmail.com>

# Conflicts:
#	README.md
#	docs/project/spec-compliance.md
…ocuments

Co-Authored-By: jason.han <hanhuijun@gmail.com>

# Conflicts:
#	README.md
#	docs/project/spec-compliance.md

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

Comment thread internal/repl/session.go
Comment on lines +629 to +641
if typed, found := s.transcript(); found {
s.ws.Open(docName, []byte(typed), s.version)
} else {
s.ws.Remove(docName)
}
live := make(map[string]bool, len(s.snippets))
for _, sn := range s.snippets {
if sn.origin == "" || sn.open {
continue
}
live[sn.origin] = true
if doc := s.ws.Document(sn.origin); doc == nil || doc.Version != sn.gen {
s.ws.Open(sn.origin, []byte(sn.src), sn.gen)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Transcript-name file drops declarations

A loaded file named <repl> shares openDocuments' transcript key. The next load or prompt submission overwrites one document, making valid declarations unreachable.

Learn more

The workspace identifies documents by name. The transcript always uses <repl>, while loaded files now use their path directly. A user can create a file literally named <repl> and load it with %load "<repl>". On the next submission, openDocuments opens both sources under the same key, so the second open replaces the first.

Example: Load %load "<repl>", then type package Typed { part def A; }. The loaded file's later open replaces the transcript document, so Typed::A no longer resolves.

Recommended fix: Give loaded documents collision-free internal workspace keys while retaining their original paths for diagnostics and display. Apply the mapping consistently to snippets, symbol DocName values, index entries, lookups, source registration, and document removal.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same finding as the earlier thread on these lines (#309 (comment)): confirmed, reachable only through a SourceFile.Name that is the literal string <repl> (every loader passes the path the file was read from; standard input is <stdin>), so on the command line it takes a file literally named <repl> in the working directory.

Closing it changes the SourceFile contract rather than the per-file migration, so it stays open for a maintainer decision between (1) documenting <repl> as the transcript's reserved name, (2) refusing a load under that name at LoadPaths/SubmitFiles, or (3) keying loaded documents apart from their names with a display name mapped back everywhere sym.DocName is shown (%explain, the debuggers, file:line:col). Left unresolved until then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant