Repository navigation
Conversation
Contributor
|
Commit: Regression test reportSelected 1 changed tests (limit=20, 0 not selected) 1 tests: regression=1 unreleased=0 coverage=0 failing=0
TestAccept/bundle/telemetry/deploy-git-origin-url-credentials/DMS=: branch ✅ | main (fa7dc9b) ❌ | latest (v1.26.0) ❌main (fa7dc9b): latest (v1.26.0): same output as main (fa7dc9b) |
pietern
commented
Nov 4, 2026
Collaborator
✅ Integration test reportCommit: 23e03dc
Top 3 slowest tests (at least 2 minutes):
|
pietern
force-pushed
the
pietern/f01-6842-telemetry-origin-url
branch
from
November 4, 2026 12:26
e28ea52 to
35adebc
Compare
idelafuente-oc
pushed a commit
to idelafuente-oc/cli
that referenced
this pull request
Nov 11, 2026
## Changes Add a `pr-review` agent skill under `.agents/skills/`. It reviews a branch (or a given commit) against its merge base with main and prints inline comments, a review body, and the evidence and worksheet behind them. It never posts. The skill works through fixed passes: correctness, duplicated mechanisms, fit with existing extension points, new error paths on absent input, sibling routes (create vs update, deploy vs destroy), user-facing names, stored state over time, doc comments, tests, and user-facing text. It reads the repo's `.agents/rules/` for testing level and comment conventions. ## Why We want agent reviews to catch the kind of issues maintainers raise in this repo, not just generic bugs, and to know how often they do. The skill was written and tuned against past reviews on this repo rather than from a checklist. ## Tests **Backtest against human reviews.** Merged PRs, each frozen at the commit that received the most human review comments, were reviewed by the skill with only read-only git access (no PR description, no GitHub), and an LLM judge checked whether it raised the same issue as each substantive human comment (match 1, partial 0.5). - 24 PRs, 96 substantive human comments, split into a tuning set (12 PRs) and a held-out set (12 PRs) whose comments were not read while writing the skill. - On held-out PRs the skill reproduces 41% of substantive comments, with 5.4 comments per PR. An earlier version that asked for three to eight comments per review reproduced 34% (range 30–38% over two runs) with 5.1 comments per PR, and a first generic version 31%. The largest gains are on PRs with many human comments, where the comment budget used to cut real findings. - Comments on tests and acceptance output are caught least often (about 28% vs 41% for source, measured on the earlier version). **Sweep of recently merged PRs.** The skill reviewed the 59 human-authored PRs merged into main between 2026-10-27 and 2026-11-03, at their final commit. It made 200 findings (about 3.4 per PR); 34 had already been raised on the PR. Each finding was then checked against the code on its own by a separate model told to try to refute it; that checker refuted all five deliberately false claims planted to test it. 193 findings were confirmed, 6 refuted and 1 left inconclusive. Of the confirmed ones not raised on the PR, 12 were bugs still present on main. Fixing them showed 2 to be working as expected; the other 10 are draft PRs databricks#7331–databricks#7340, each naming the PR that introduced the issue. The refuted findings were mostly behavior that the code or a test already documented as intended. The sweep was run with the earlier version that asked for three to eight comments per review. Each review takes 2–5 minutes with Opus. The backtest harness and fixtures are kept outside this repo so future runs can't read the expected comments. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
bundleGitInfo assumed load_git_details had already stripped userinfo from bundle.git.origin_url, but that only happens on the local .git route. A URL set in databricks.yml, or returned by the workspace API for a Git folder, was sent to telemetry as written, including any embedded token. Strip userinfo when building the telemetry git info so every route is covered. Extract the stripping from Repository.OriginUrl into git.StripURLCredentials and use it for telemetry too. Co-authored-by: Isaac <no-reply@databricks.com>
pietern
force-pushed
the
pietern/f01-6842-telemetry-origin-url
branch
from
November 16, 2026 10:13
35adebc to
23e03dc
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Strip credentials from the git origin URL before it goes into deploy telemetry.
bundleGitInfoassumed the URL was already stripped, but only the local.gitroute does that; a URL frombundle.git.origin_urlor the workspace API went through as written. The stripping inRepository.OriginUrl()moves intogit.StripURLCredentials, which telemetry now uses too.DMS and deploy metadata still receive the value as written; left for a follow-up.
Why
#6842 added the origin URL to deploy telemetry, assuming its credentials were already stripped.
Tests
TestStripURLCredentialsinlibs/gitand a wiring test inbundle/phases.bundle/telemetry/deploy-git-origin-url-credentialsshows the token stripped.This pull request and its description were written by Isaac.