-
Notifications
You must be signed in to change notification settings - Fork 269
[Fix] Task history can disappear when users restart after completion #1452
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
zoomote
wants to merge
27
commits into
main
Choose a base branch
from
fix/model-completion-persistence-176wyz2fscjpe
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
27 commits
Select commit
Hold shift + click to select a range
a77af15
test(formal): model completion persistence ordering
roomote 0cbee54
test(task): reproduce completion persistence race
roomote a5ee06e
fix(task): persist history before completion
roomote 2e7d094
test(e2e): require restored completion turn
roomote b70f20c
test(api): strengthen completion persistence checks
roomote 8057e76
fix(task): cancel pending persistence waits
roomote 22884ed
fix(task): stop persistence retries on cancel
roomote 2941ec5
test(task): cover completion retry recovery
roomote b1c5711
fix: apply CodeRabbit auto-fixes
coderabbitai[bot] 79dc9e6
fix(task): keep persistence retry timers generation-local
roomote 993fcec
test(formal): model completion persistence lifecycle
roomote 1d6f49f
test(formal): model delegated completion ordering
roomote cd35700
test(task): satisfy changed-code mutation gate
roomote b5afb17
refactor(task): remove duplicate cancellation branches
roomote 0833723
test(task): cover same-turn persistence cancellation
roomote eee53b8
fix(task): suppress completion after durable cancellation
roomote bf409c8
refactor(task): unify persistence cancellation results
roomote 096595e
refactor(task): derive readiness from generation state
roomote ec75de8
test(formal): compose persistence into model check
roomote 9b66621
fix(api): emit delegated completion after child disposal
roomote b487d8e
fix(api): emit delegated completion after child disposal
roomote ea61cf1
test(api): cover delegated completion forwarding
roomote 09f0255
fix(ci): preserve lifecycle model command
roomote d5a18ac
fix(task): retry assistant persistence before tool results
roomote 69719ae
test(task): cover cancelled tool-result flush
roomote 66b7b0a
chore(task): align persistence checks with cleanup rebase
roomote 519813f
fix(e2e): replace undefined conversationLength with sequence check
edelauna File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,288 @@ | ||
| type TaskKind = "standalone" | "delegated" | ||
| type HistoryPhase = "idle" | "writing" | "failed" | "durable" | "exhausted" | ||
| type RetryPhase = "idle" | "waiting" | "ready" | ||
| type WriteStarts = 0 | 1 | 2 | ||
| type DelegationPhase = "not-applicable" | "awaiting-reopen" | "reopened" | "reopen-failed" | ||
|
|
||
| interface ModelState { | ||
| kind: TaskKind | ||
| history: HistoryPhase | ||
| retry: RetryPhase | ||
| writeStarts: WriteStarts | ||
| completionAccepted: boolean | ||
| completionEmitted: boolean | ||
| cancelled: boolean | ||
| waitSettled: boolean | ||
| cancelledAtRetryBoundary: boolean | ||
| delegation: DelegationPhase | ||
| } | ||
|
|
||
| interface Transition { | ||
| name: string | ||
| next: ModelState | ||
| } | ||
|
|
||
| interface TraceStep { | ||
| action: string | ||
| state: ModelState | ||
| } | ||
|
|
||
| const MAX_DEPTH = 10 | ||
| const MAX_STATES = 1_000 | ||
| const taskKinds = ["standalone", "delegated"] as const | ||
| const expectedActions = [ | ||
| "start-initial-write", | ||
| "accept-completion", | ||
| "finish-write", | ||
| "fail-write", | ||
| "schedule-retry", | ||
| "finish-retry-delay", | ||
| "start-retry-write", | ||
| "exhaust-retries", | ||
| "cancel", | ||
| "reopen-parent", | ||
| "fail-parent-reopen", | ||
| "emit-completion", | ||
| ] as const | ||
| const stateInvariants = { | ||
| "completion requires accepted restart-visible history": (state: ModelState) => | ||
| state.completionEmitted && (!state.completionAccepted || state.history !== "durable" || !state.waitSettled) | ||
| ? "completion emitted before accepted assistant history became restart-visible" | ||
| : undefined, | ||
| "delayed and failed persistence keep completion pending": (state: ModelState) => { | ||
| if (state.cancelled || !state.completionAccepted) return undefined | ||
| if ((state.history === "writing" || state.history === "failed") && state.waitSettled) { | ||
| return "completion wait settled while persistence could still retry" | ||
| } | ||
| if (state.history === "exhausted" && (!state.waitSettled || state.completionEmitted)) { | ||
| return "exhausted persistence did not settle without completion" | ||
| } | ||
| return state.history !== "durable" && state.completionEmitted | ||
| ? "delayed or failed persistence allowed completion" | ||
| : undefined | ||
| }, | ||
| "cancellation settles waits and suppresses retry/completion": (state: ModelState) => | ||
| state.cancelled && (!state.waitSettled || state.retry !== "idle" || state.completionEmitted) | ||
| ? "cancellation did not settle the wait and suppress retry/completion" | ||
| : undefined, | ||
| "delegated completion requires successful parent reopen": (state: ModelState) => | ||
| state.kind === "delegated" && state.completionEmitted && state.delegation !== "reopened" | ||
| ? "delegated completion emitted before the parent reopened" | ||
| : undefined, | ||
| } satisfies Record<string, (state: ModelState) => string | undefined> | ||
| const transitionInvariants = { | ||
| "cancellation starts no later write or completion": (previous: ModelState, transition: Transition) => { | ||
| if (previous.cancelled && transition.next.writeStarts > previous.writeStarts) { | ||
| return `cancelled task started a stale history write after ${transition.name}` | ||
| } | ||
| if (previous.cancelled && !previous.completionEmitted && transition.next.completionEmitted) { | ||
| return `cancelled task emitted completion after ${transition.name}` | ||
| } | ||
| return undefined | ||
| }, | ||
| } satisfies Record<string, (previous: ModelState, transition: Transition) => string | undefined> | ||
| const semanticLandmarks = { | ||
| "delayed-completion-pending": (state: ModelState) => | ||
| state.completionAccepted && state.history === "writing" && !state.completionEmitted, | ||
| "failed-completion-pending": (state: ModelState) => | ||
| state.completionAccepted && state.history === "failed" && !state.completionEmitted, | ||
| "exhausted-completion-pending": (state: ModelState) => | ||
| state.completionAccepted && state.history === "exhausted" && state.waitSettled && !state.completionEmitted, | ||
| "cancelled-retry-boundary": (state: ModelState) => | ||
| state.cancelledAtRetryBoundary && state.waitSettled && state.retry === "idle" && !state.completionEmitted, | ||
| "standalone-durable-completion": (state: ModelState) => | ||
| state.kind === "standalone" && state.history === "durable" && state.completionEmitted, | ||
| "delegated-durable-completion": (state: ModelState) => | ||
| state.kind === "delegated" && | ||
| state.history === "durable" && | ||
| state.delegation === "reopened" && | ||
| state.completionEmitted, | ||
| "delegated-reopen-failure-pending": (state: ModelState) => | ||
| state.kind === "delegated" && state.delegation === "reopen-failed" && !state.completionEmitted, | ||
| } satisfies Record<string, (state: ModelState) => boolean> | ||
|
|
||
| function initialState(kind: TaskKind): ModelState { | ||
| return { | ||
| kind, | ||
| history: "idle", | ||
| retry: "idle", | ||
| writeStarts: 0, | ||
| completionAccepted: false, | ||
| completionEmitted: false, | ||
| cancelled: false, | ||
| waitSettled: false, | ||
| cancelledAtRetryBoundary: false, | ||
| delegation: kind === "delegated" ? "awaiting-reopen" : "not-applicable", | ||
| } | ||
| } | ||
|
|
||
| function transitions(state: ModelState): Transition[] { | ||
| const result: Transition[] = [] | ||
|
|
||
| if (state.history === "idle" && !state.cancelled) { | ||
| result.push({ | ||
| name: "start-initial-write", | ||
| next: { ...state, history: "writing", writeStarts: 1 }, | ||
| }) | ||
| } | ||
| if (!state.completionAccepted && !state.cancelled) { | ||
| result.push({ name: "accept-completion", next: { ...state, completionAccepted: true } }) | ||
| } | ||
| if (state.history === "writing") { | ||
| result.push({ | ||
| name: "finish-write", | ||
| next: { ...state, history: "durable", waitSettled: true }, | ||
| }) | ||
| result.push({ name: "fail-write", next: { ...state, history: "failed" } }) | ||
| } | ||
| if (state.history === "failed" && state.retry === "idle" && !state.cancelled) { | ||
| if (state.writeStarts < 2) { | ||
| result.push({ name: "schedule-retry", next: { ...state, retry: "waiting" } }) | ||
| } else { | ||
| result.push({ | ||
| name: "exhaust-retries", | ||
| next: { ...state, history: "exhausted", waitSettled: true }, | ||
| }) | ||
| } | ||
| } | ||
| if (state.retry === "waiting" && !state.cancelled) { | ||
| result.push({ name: "finish-retry-delay", next: { ...state, retry: "ready" } }) | ||
| } | ||
| if (state.retry === "ready" && !state.cancelled && state.writeStarts < 2) { | ||
| result.push({ | ||
| name: "start-retry-write", | ||
| next: { | ||
| ...state, | ||
| history: "writing", | ||
| retry: "idle", | ||
| writeStarts: (state.writeStarts + 1) as WriteStarts, | ||
| }, | ||
| }) | ||
| } | ||
| if (!state.cancelled && !state.completionEmitted) { | ||
| result.push({ | ||
| name: "cancel", | ||
| next: { | ||
| ...state, | ||
| retry: "idle", | ||
| cancelled: true, | ||
| waitSettled: true, | ||
| cancelledAtRetryBoundary: state.retry === "ready", | ||
| }, | ||
| }) | ||
| } | ||
| if ( | ||
| state.kind === "delegated" && | ||
| state.delegation === "awaiting-reopen" && | ||
| state.completionAccepted && | ||
| state.history === "durable" && | ||
| state.waitSettled && | ||
| !state.cancelled | ||
| ) { | ||
| result.push({ name: "reopen-parent", next: { ...state, delegation: "reopened" } }) | ||
| result.push({ name: "fail-parent-reopen", next: { ...state, delegation: "reopen-failed" } }) | ||
| } | ||
| if ( | ||
| state.completionAccepted && | ||
| state.history === "durable" && | ||
| state.waitSettled && | ||
| (state.kind === "standalone" || state.delegation === "reopened") && | ||
| !state.completionEmitted && | ||
| !state.cancelled | ||
| ) { | ||
| result.push({ | ||
| name: "emit-completion", | ||
| next: { ...state, completionEmitted: true, waitSettled: true }, | ||
| }) | ||
| } | ||
|
|
||
| return result | ||
| } | ||
|
|
||
| function invariantViolations(state: ModelState): string[] { | ||
| return Object.entries(stateInvariants).flatMap(([name, check]) => { | ||
| const violation = check(state) | ||
| return violation ? [`${name}: ${violation}`] : [] | ||
| }) | ||
| } | ||
|
|
||
| function transitionViolations(previous: ModelState, transition: Transition): string[] { | ||
| return Object.entries(transitionInvariants).flatMap(([name, check]) => { | ||
| const violation = check(previous, transition) | ||
| return violation ? [`${name}: ${violation}`] : [] | ||
| }) | ||
| } | ||
|
|
||
| function canonical(state: ModelState): string { | ||
| return JSON.stringify(state) | ||
| } | ||
|
|
||
| function formatCounterexample(message: string, trace: TraceStep[]): string { | ||
| return [ | ||
| `Completion persistence invariant failed: ${message}`, | ||
| `Bounds: depth=${MAX_DEPTH}, states=${MAX_STATES}, writes<=2`, | ||
| ...trace.map((step, index) => `${index}. ${step.action}\n${JSON.stringify(step.state, null, 2)}`), | ||
| ].join("\n") | ||
| } | ||
|
|
||
| function runModelCheck(): number { | ||
| const queue: Array<{ state: ModelState; trace: TraceStep[] }> = taskKinds.map((kind) => { | ||
| const state = initialState(kind) | ||
| return { state, trace: [{ action: `initial(${kind})`, state }] } | ||
| }) | ||
| const visited = new Set(queue.map(({ state }) => canonical(state))) | ||
| const reachedActions = new Set<string>() | ||
| const reachedLandmarks = new Set<string>() | ||
| const frontier: ModelState[] = [] | ||
|
|
||
| for (let index = 0; index < queue.length; index++) { | ||
| const node = queue[index]! | ||
| for (const [name, predicate] of Object.entries(semanticLandmarks)) { | ||
| if (predicate(node.state)) reachedLandmarks.add(name) | ||
| } | ||
| const violations = invariantViolations(node.state) | ||
| if (violations.length) throw new Error(formatCounterexample(violations.join("; "), node.trace)) | ||
| if (node.trace.length - 1 === MAX_DEPTH) { | ||
| frontier.push(node.state) | ||
| continue | ||
| } | ||
|
|
||
| for (const transition of transitions(node.state)) { | ||
| reachedActions.add(transition.name) | ||
| const trace = [...node.trace, { action: transition.name, state: transition.next }] | ||
| const violations = transitionViolations(node.state, transition) | ||
| if (violations.length) throw new Error(formatCounterexample(violations.join("; "), trace)) | ||
| const key = canonical(transition.next) | ||
| if (visited.has(key)) continue | ||
| visited.add(key) | ||
| queue.push({ state: transition.next, trace }) | ||
| if (visited.size > MAX_STATES) { | ||
| throw new Error(`Completion persistence exploration exceeded its ${MAX_STATES}-state budget`) | ||
| } | ||
| } | ||
| } | ||
|
|
||
| const unreachableActions = expectedActions.filter((action) => !reachedActions.has(action)) | ||
| if (unreachableActions.length) { | ||
| throw new Error(`Completion persistence model has unreachable actions: ${unreachableActions.join(", ")}`) | ||
| } | ||
| const missingLandmarks = Object.keys(semanticLandmarks).filter((name) => !reachedLandmarks.has(name)) | ||
| if (missingLandmarks.length) { | ||
| throw new Error(`Completion persistence model has unreachable landmarks: ${missingLandmarks.join(", ")}`) | ||
| } | ||
| const unexploredSuccessor = frontier | ||
| .flatMap((state) => transitions(state)) | ||
| .find((transition) => !visited.has(canonical(transition.next))) | ||
| if (unexploredSuccessor) { | ||
| throw new Error( | ||
| `Completion persistence exploration reached depth ${MAX_DEPTH} with an unseen successor (${unexploredSuccessor.name})`, | ||
| ) | ||
| } | ||
| return visited.size | ||
| } | ||
|
|
||
| const checkedStates = runModelCheck() | ||
| const invariantCount = Object.keys(stateInvariants).length + Object.keys(transitionInvariants).length | ||
| console.log( | ||
| `Completion persistence model check passed: ${checkedStates} states, ${expectedActions.length}/${expectedActions.length} actions reachable, ${invariantCount} invariants, ${Object.keys(semanticLandmarks).length}/${Object.keys(semanticLandmarks).length} landmarks reached, depth <= ${MAX_DEPTH}, writes <= 2`, | ||
| ) |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Require
hasTaskApiConversationHistorySequenceto check the adjacent assistant turn.The helper uses
.slice(userTurnIndex + 1).some(...), so a later matchingattempt_completioncan satisfy both restart assertions even when the next persisted turn is missing or reordered. Check onlyapiConversationHistory[userTurnIndex + 1], and compare the marker value exactly. Add a boundary test for an intervening turn.🤖 Prompt for AI Agents