Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions evalboard/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -207,5 +207,15 @@ belongs to it.
and then every 4 minutes (`lib/warm.ts`, started from `instrumentation.ts`), so
the per-run cache is rebuilt by those requests rather than by a visitor.
`EVALBOARD_WARM_INTERVAL_SECONDS` sets the interval; `0` turns it off.
- Task Failure Analysis (internal edition, skills source): the run page reads
the skill-doctor report under `<run>/skill-doctor/` (`lib/doctor.ts`,
`readRunDoctor`). `status.json` is refetched after 60 s; the triage file it
names, `<fixes_dir>/fixes.json` and up to 10 patches are cached for good.
`/api/file` serves only the patches fixes.json lists, loading them through
the same reader on a cold cache. The report is a collapsed block
under the AI summary, laid out like the doctor's own page (stats, failures by
layer, Fix first, per-skill failure cards). When the report is done and
`analysis.md` has a `## Findings` line, the AI summary stops there; other
states keep the AI summary and show the doctor's state in the block header. `EVALBOARD_TASK_ANALYSIS=off` turns it off.
- Pass rows render green (`bg-green-50 text-green-700`), failures render red
(`bg-red-50 text-red-700`), on a white background.
675 changes: 675 additions & 0 deletions evalboard/app/runs/[id]/__tests__/task-failure-analysis.test.tsx

Large diffs are not rendered by default.

25 changes: 23 additions & 2 deletions evalboard/app/runs/[id]/page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import {
findMatureSourceRuns,
readActivationScore,
readRunAnalysis,
readRunDoctor,
readRunMeta,
readRunSummary,
readRunTasks,
Expand All @@ -24,6 +25,8 @@ import { sourceById } from "@/lib/sources";
import { scalarParam } from "@/app/_lib/source-param";
import { fmtRunTime } from "@/lib/format";
import { AnalysisPanel } from "./analysis-panel";
import { TaskFailureAnalysis } from "./task-failure-analysis";
import { planAnalysis, taskAnalysisEnabled } from "@/lib/doctor";
import { RefreshButton } from "./refresh-button";
import { RunView } from "./run-view";
import { RunIdentity } from "./run-identity";
Expand All @@ -43,7 +46,14 @@ export default async function RunPage({
// Which container this run lives in. Unknown/absent coerces to the skills
// nightly, so every URL that predates the Scribe tab keeps resolving as-is.
const source = sourceById(scalarParam((await searchParams).src));
const [summary, tasks, activation, analysis, reviewIndex, meta] =
// The skill-doctor report: internal-only, skills nightly only, with an env
// kill switch (EVALBOARD_TASK_ANALYSIS=off). readRunDoctor never throws.
const doctorOn = taskAnalysisEnabled(
isInternal,
source.id,
process.env.EVALBOARD_TASK_ANALYSIS,
);
const [summary, tasks, activation, analysis, reviewIndex, meta, doctor] =
await Promise.all([
readRunSummary(id, source),
readRunTasks(id, source),
Expand All @@ -53,6 +63,7 @@ export default async function RunPage({
readRunAnalysis(id, source),
readRunReviewIndex(id, source),
readRunMeta(id, source),
doctorOn ? readRunDoctor(id, source) : Promise.resolve(null),
]);
if (!summary || !tasks) notFound();
// Mature-skipped tasks weren't executed this run, so they have no detail page
Expand Down Expand Up @@ -85,6 +96,9 @@ export default async function RunPage({
const tagCounts = reviewIndex ? tagCountsForRun(reviewIndex) : [];
const reviewsByTask = reviewIndex ? indexByTask(reviewIndex) : undefined;
const human = fmtRunTime(id);
// A done report replaces analysis.md's Findings and everything below it;
// otherwise the AI summary is unchanged. The doctor block sits under it.
const plan = planAnalysis(analysis, doctor);
const series = isInternal
? seriesForRun(source.id, {
series: meta?.series ?? null,
Expand Down Expand Up @@ -156,7 +170,14 @@ export default async function RunPage({
)}
</div>

{analysis && <AnalysisPanel markdown={analysis} />}
{plan.summary && <AnalysisPanel markdown={plan.summary} />}
{plan.doctor && (
<TaskFailureAnalysis
doctor={plan.doctor}
runId={id}
sourceId={source.id}
/>
)}

{/* The header above streams as soon as run.json is parsed, but the
grid waits on the review index and the mature-source scan. With
Expand Down
Loading
Loading