Skip to content

aitools telemetry: attach the calling agent to the install event - #7415

Closed
anton-107 wants to merge 2 commits into
mainfrom
aitools-calling-agent
Closed

anton-107 wants to merge 2 commits into
mainfrom
aitools-calling-agent

Conversation

@anton-107

Copy link
Copy Markdown
Contributor

Summary

aitools install telemetry now records which coding agent ran the CLI, as a closed enum.

 logInstallEvent
   AitoolsInstallEvent{
     Agents:        [install targets]
     AgentResults:  ...
+    CallingAgent:  agents.CallingAgentType()
   }
CallingAgentType()                       # libs/aitools/agents/calling.go
  provider := useragent.AgentProvider()  # SDK env-var detection
  ""              -> NONE                # a human or a script
  "multiple"      -> MULTIPLE            # an agent running another agent
  registry agent  -> CLAUDE_CODE, CURSOR, ... (the 10 agents the CLI installs into)
  SDK-only agent  -> AMP, AUGMENT, CLINE, OPENCLAW, VSCODE_AGENT, WINDSURF
  anything else   -> UNKNOWN             # free-text AGENT / AI_AGENT is never sent verbatim
 libs/aitools/agents/
+├── calling.go              # SDK agent -> registry agent -> telemetry enum (moved from cmd/aitools agentType)
+├── calling_test.go
+└── agenttest/agenttest.go  # SetAgentEnv: pins the detected agent in tests
 libs/telemetry/protos/aitools_install.go  # +9 enum values, +calling_agent field

The SDK's agent list (listKnownAgents) is unexported, so agenttest.agentEnvVars and the product list in TestProviderAgentType are copied by hand. A comment at each says what to update on an SDK bump. Until someone updates them, an agent that only a newer SDK detects is sent as UNKNOWN.

Evidence

TestLogInstallEventSendsCallingAgent runs logInstallEvent, uploads through telemetry.Upload to a local /telemetry-ext test server, and reads calling_agent from the payload.

  • Before (CallingAgent: assignment removed):
    --- FAIL: TestLogInstallEventSendsCallingAgent/claude_code   expected: "CLAUDE_CODE"  actual: ""
    --- FAIL: TestLogInstallEventSendsCallingAgent/no_agent      expected: "NONE"         actual: ""
    
    After:
    --- PASS: TestLogInstallEventSendsCallingAgent/claude_code
    --- PASS: TestLogInstallEventSendsCallingAgent/no_agent
    ok  	github.com/databricks/cli/cmd/aitools
    

The test clears DATABRICKS_CLI_DISABLE_TELEMETRY / DATABRICKS_CLI_TELEMETRY_TIMEOUT and pins PAT auth, so the runner's environment does not affect it. Also run: go test for libs/aitools/..., cmd/aitools, libs/telemetry/..., and the aitools acceptance tests.

Merge Danger

Door: two-way

Adds one telemetry field and some enum values. A revert removes them. Events that were already sent keep the field.

Blast Radius: telemetry-only

This changes only the payload of the aitools install event. Install behaviour does not change. Agent names are sent only as enum values, never as raw env-var text.

This pull request and its description were written by Isaac.

Adds a closed AitoolsAgentType enum for the agent calling the CLI (the 10
registry agents, the 6 SDK-only agents, plus UNKNOWN / MULTIPLE / NONE),
the SDK agent -> registry mapping in libs/aitools/agents, and sets
calling_agent on AitoolsInstallEvent. Free-text AGENT / AI_AGENT values
are never sent.

Co-authored-by: Isaac <no-reply@databricks.com>
@anton-107
anton-107 requested review from a team and lennartkats-db as code owners November 9, 2026 16:12
@anton-107 anton-107 changed the title aitools: attach the calling agent to the install event aitools telemetry: attach the calling agent to the install event Nov 9, 2026
@anton-107
anton-107 enabled auto-merge November 9, 2026 16:14
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 6b14bd3

Run: 42652898142

Env 🔄​flaky ✅​pass 🙈​skip Time
🔄​ aws linux-2core-8gb 2 269 15 4:29
✅​ aws-windows-latest-4core-16gb 273 13 3:11
✅​ azure linux-2core-8gb 270 15 3:55
✅​ azure-windows-latest-4core-16gb 272 13 3:11
✅​ gcp linux-2core-8gb 271 15 4:06
✅​ gcp-windows-latest-4core-16gb 273 13 3:14
Test Name aws linux-2core-8gb
🔄​ TestFilerWorkspaceNotebook 🔄​f
🔄​ TestFilerWorkspaceNotebook/scalaJupyterNb.ipynb 🔄​f
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:09 gcp-windows-latest-4core-16gb TestAccept
3:06 aws-windows-latest-4core-16gb TestAccept
3:05 azure-windows-latest-4core-16gb TestAccept

@anton-107 anton-107 closed this Nov 10, 2026
auto-merge was automatically disabled November 10, 2026 09:14

Pull request was closed

@anton-107

Copy link
Copy Markdown
Contributor Author

closed as unnecessary

@simonfaltum simonfaltum left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking issues found in the reviewed commit. The focused agent, telemetry, and aitools tests passed. The inline comments are nonblocking documentation and cleanup suggestions.


// ProviderAgentType maps an SDK agent product to the closed telemetry enum.
// Free text from the AGENT / AI_AGENT env vars, and products a newer SDK
// detects, become UNKNOWN, so the raw value never leaves the machine.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nice to have] This comment overstates the guarantee: AGENT=cursor becomes CURSOR, and an unknown AGENT value still travels in the SDK's HTTP User-Agent. Please say that recognized product names map to enums and other values to UNKNOWN, so this JSON field never contains free-form text.

// process-wide AgentProvider cache before and after the test. With no
// arguments AgentProvider() reports no agent. Tests using it must not run in
// parallel.
func SetAgentEnv(t *testing.T, kv ...string) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nice to have] internal/testutil.SetAgent already clears the environment and resets the SDK's agent cache. These tests can use testutil.SetAgent(t, "") followed by the relevant t.Setenv calls, avoiding another manually maintained SDK environment-variable list.

// CallingAgent returns the registry agent driving the CLI, or nil when no
// agent is detected, several are, or the agent is one the CLI cannot install
// into.
func CallingAgent() *Agent {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nice to have] CallingAgent() currently has only test callers, and AgentForProvider has no production consumer outside this package. Consider dropping CallingAgent and keeping the provider lookup unexported until there is a production caller.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants