Repository navigation
Add pr-review agent skill - #7344
Merged
Merged
Conversation
Reviews a branch or commit against its merge base with main as a maintainer would: correctness, duplicated mechanisms, fit with existing extension points, error paths on absent input, sibling routes, user-facing names and text, stored state over time, doc comments, and tests. It prints comments with the evidence behind them and never posts. Co-authored-by: Isaac <no-reply@databricks.com>
Collaborator
Integration test reportCommit: 552a987
Top 3 slowest tests (at least 2 minutes):
|
Drop the three-to-eight comment budget, which made the skill leave out confirmed findings on larger changes. Each comment now has to state the behavior, a concrete consequence and an ask; comments are ordered by severity; and a final coverage check lists any worksheet row that is not raised, with the reason. Co-authored-by: Isaac <no-reply@databricks.com>
shreyas-goenka
approved these changes
Nov 5, 2026
idelafuente-oc
pushed a commit
to idelafuente-oc/cli
that referenced
this pull request
Nov 11, 2026
…databricks#7386) ## Changes Adds one bullet to pass 9 (Tests) of `.agents/skills/pr-review/SKILL.md`, placed after the bullet on whether each new or changed test shows the effect it claims. When a change sends a new field, value format or API call, and no covering test is `Cloud = true` and the description does not mention a real-workspace run, the review asks whether it was verified against one. Nothing else in the skill changes. No `.nextchanges/` fragment is added, matching databricks#7344, which added the skill. ## Why The fake server in `libs/testserver` only returns what its handlers were taught. A test that records the request, or a handler that echoes a field back, shows what the CLI sends, not what the API accepts. Pass 9 never asked whether such a change had been checked against a real workspace, and maintainers ask that question often. The gap showed up in a review of databricks#7184 ("Support file selectors in pipelines run"). The change started sending new `refresh_selection_by_path` values (workspace paths, notebook extension stripped) to the pipelines StartUpdate API. The only coverage was the fake server echoing them back. The test was not `Cloud = true`, and the PR description said no live-workspace run was done. The first review question was whether it had been run against a real workspace, and the skill as written would not have raised it. The bullet asks a question rather than requiring a cloud run. Every acceptance test still runs locally, and `Cloud = true` only adds a run against a real workspace. ## Tests - Markdown-only change. No Go code or acceptance tests changed. - Ran the whitespace validator that `./task ws` runs, without `--fix`: 10787 checked, 0 failed. - Checked the bullet's claims against `.agents/rules/testing.md`: `Cloud = true` adds a cloud run and never removes the local one, `Cloud` is inherited (so `out.test.toml` holds the effective value), and the fake server lives in `libs/testserver`. This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
tedjt
pushed a commit
to tedjt/cli
that referenced
this pull request
Nov 12, 2026
…7484) ## Changes Follow-up to databricks#7483. `testmask` now ignores agent config (`.agents/`, `.claude/`, `.cursor/`, `AGENTS.md`), root docs (`README.md`, `CHANGELOG.md`, `SECURITY.md`), GitHub issue/PR templates, and `.nextchanges` fragments. Ignored entries support `path.Match` globs and a trailing `/**` for directories. ## Why PRs touching only these files (e.g. databricks#7464, databricks#7448, databricks#7344) still ran the full test matrix. Nested READMEs are left out since many are acceptance outputs or embedded templates. ## Tests Added unit cases, and testmask returns `[]` for the PRs above. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Add a
pr-reviewagent skill under.agents/skills/. It reviews a branch (or a given commit) against its merge base with main and prints inline comments, a review body, and the evidence and worksheet behind them. It never posts.The skill works through fixed passes: correctness, duplicated mechanisms, fit with existing extension points, new error paths on absent input, sibling routes (create vs update, deploy vs destroy), user-facing names, stored state over time, doc comments, tests, and user-facing text. It reads the repo's
.agents/rules/for testing level and comment conventions.Why
We want agent reviews to catch the kind of issues maintainers raise in this repo, not just generic bugs, and to know how often they do. The skill was written and tuned against past reviews on this repo rather than from a checklist.
Tests
Backtest against human reviews. Merged PRs, each frozen at the commit that received the most human review comments, were reviewed by the skill with only read-only git access (no PR description, no GitHub), and an LLM judge checked whether it raised the same issue as each substantive human comment (match 1, partial 0.5).
Sweep of recently merged PRs. The skill reviewed the 59 human-authored PRs merged into main between 2026-10-27 and 2026-11-03, at their final commit. It made 200 findings (about 3.4 per PR); 34 had already been raised on the PR. Each finding was then checked against the code on its own by a separate model told to try to refute it; that checker refuted all five deliberately false claims planted to test it. 193 findings were confirmed, 6 refuted and 1 left inconclusive. Of the confirmed ones not raised on the PR, 12 were bugs still present on main. Fixing them showed 2 to be working as expected; the other 10 are draft PRs #7331–#7340, each naming the PR that introduced the issue. The refuted findings were mostly behavior that the code or a test already documented as intended.
The sweep was run with the earlier version that asked for three to eight comments per review.
Each review takes 2–5 minutes with Opus. The backtest harness and fixtures are kept outside this repo so future runs can't read the expected comments.
This pull request and its description were written by Isaac.