Repository navigation
Conversation
The report-failure step ran on any failure, so a failed workflow_dispatch run (for example a mistyped version input in bump-go-toolchain, or a non-dry-run manual run of bump-vuln-deps) opened an issue claiming the scheduled workflow failed and assigned seven people. Gate the step on github.event_name == 'schedule' in both workflows; this subsumes the !inputs.dry_run check in bump-vuln-deps. Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
Integration test reportCommit: 2880490
Top 3 slowest tests (at least 2 minutes):
|
Contributor
Author
|
For manual testing of the workflow we want to create an issue on workflow_dispatch. |
idelafuente-oc
pushed a commit
to idelafuente-oc/cli
that referenced
this pull request
Nov 11, 2026
## Changes Add a `pr-review` agent skill under `.agents/skills/`. It reviews a branch (or a given commit) against its merge base with main and prints inline comments, a review body, and the evidence and worksheet behind them. It never posts. The skill works through fixed passes: correctness, duplicated mechanisms, fit with existing extension points, new error paths on absent input, sibling routes (create vs update, deploy vs destroy), user-facing names, stored state over time, doc comments, tests, and user-facing text. It reads the repo's `.agents/rules/` for testing level and comment conventions. ## Why We want agent reviews to catch the kind of issues maintainers raise in this repo, not just generic bugs, and to know how often they do. The skill was written and tuned against past reviews on this repo rather than from a checklist. ## Tests **Backtest against human reviews.** Merged PRs, each frozen at the commit that received the most human review comments, were reviewed by the skill with only read-only git access (no PR description, no GitHub), and an LLM judge checked whether it raised the same issue as each substantive human comment (match 1, partial 0.5). - 24 PRs, 96 substantive human comments, split into a tuning set (12 PRs) and a held-out set (12 PRs) whose comments were not read while writing the skill. - On held-out PRs the skill reproduces 41% of substantive comments, with 5.4 comments per PR. An earlier version that asked for three to eight comments per review reproduced 34% (range 30–38% over two runs) with 5.1 comments per PR, and a first generic version 31%. The largest gains are on PRs with many human comments, where the comment budget used to cut real findings. - Comments on tests and acceptance output are caught least often (about 28% vs 41% for source, measured on the earlier version). **Sweep of recently merged PRs.** The skill reviewed the 59 human-authored PRs merged into main between 2026-10-27 and 2026-11-03, at their final commit. It made 200 findings (about 3.4 per PR); 34 had already been raised on the PR. Each finding was then checked against the code on its own by a separate model told to try to refute it; that checker refuted all five deliberately false claims planted to test it. 193 findings were confirmed, 6 refuted and 1 left inconclusive. Of the confirmed ones not raised on the PR, 12 were bugs still present on main. Fixing them showed 2 to be working as expected; the other 10 are draft PRs databricks#7331–databricks#7340, each naming the PR that introduced the issue. The refuted findings were mostly behavior that the code or a test already documented as intended. The sweep was run with the earlier version that asked for three to eight comments per review. Each review takes 2–5 minutes with Opus. The backtest harness and fixtures are kept outside this repo so future runs can't read the expected comments. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
The "Report failure" step in
bump-go-toolchain.yml(if: failure()) andbump-vuln-deps.yml(if: failure() && !inputs.dry_run) also ran for manualworkflow_dispatchruns, so a mistypedversioninput opened a "the scheduled workflow failed" issue and assigned seven people. Both now useif: failure() && github.event_name == 'schedule'; thereport-failureaction's description recommends the same gate.Question: in
bump-vuln-deps.ymlthis drops!inputs.dry_run, so a failed manual non-dry-run no longer reports either, and the reporter can't be tested by dispatching a run by hand. Is schedule-only what we want?Why
Follow-up to #7312, which added the failure reporter.
Tests
This pull request and its description were written by Isaac.