Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
ca772f4
Log AitoolsMcpEvent from aitools mcp and the serve proxy
simonfaltum Nov 12, 2026
635ce6d
Create the agent config dirs in the telemetry acceptance cases
simonfaltum Nov 12, 2026
b6f82a6
Record the sign-in prompt only when a page URL was shown
simonfaltum Nov 13, 2026
231e0b4
Keep one set of login steps on an MCP authentication failure
simonfaltum Nov 13, 2026
a87a80d
Resolve the remove upload's auth after the local edit
simonfaltum Nov 13, 2026
3452750
Log serve's start after its checks, and keep the failed start's upload
simonfaltum Nov 13, 2026
2bd5d17
Log only events with a reached server, and count profiles by config file
simonfaltum Nov 13, 2026
0f3e651
Derive the telemetry service enum from the known system.ai table
simonfaltum Nov 13, 2026
61fdc2c
Tidy the telemetry acceptance scripts and the fake telemetry endpoint
simonfaltum Nov 13, 2026
48880f0
Document the error categories of the server result and tidy its test
simonfaltum Nov 13, 2026
b50fd13
Authenticate telemetry uploads without refreshing a token
simonfaltum Nov 13, 2026
b5918d0
Resolve remove's telemetry auth only after it logs an event
simonfaltum Nov 13, 2026
8c93399
Resolve add, login and logout telemetry auth as install does
simonfaltum Nov 13, 2026
76a784d
Log PROXY_STARTED only for a serving start
simonfaltum Nov 13, 2026
e98a36a
Tidy the MCP telemetry: sign-in and record-only states, service enum,…
simonfaltum Nov 13, 2026
2ab75de
Upload telemetry with a config the command sets
simonfaltum Nov 13, 2026
8a5baa2
Authenticate MCP telemetry uploads with the profile each command binds
simonfaltum Nov 13, 2026
a40ef5c
Record sign-in prompts of waits on a cancelled page, and test the ref…
simonfaltum Nov 13, 2026
be0e3ba
Tidy the MCP telemetry: one SIGN_IN builder, rejected-start test, doc…
simonfaltum Nov 13, 2026
1174ea7
Refuse the MCP telemetry upload after the service rejects the credential
simonfaltum Nov 13, 2026
a1d30e0
Keep the telemetry upload's host metadata failures out of the command…
simonfaltum Nov 13, 2026
f20df6b
Group the MCP telemetry profile count as mcp list does, and tidy the …
simonfaltum Nov 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions acceptance/cmd/aitools/telemetry/mcp-add/.databrickscfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[logfood]
host = $DATABRICKS_HOST
token = test-token-logfood
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
not = [valid
1 change: 1 addition & 0 deletions acceptance/cmd/aitools/telemetry/mcp-add/out.test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Cloud = false
94 changes: 94 additions & 0 deletions acceptance/cmd/aitools/telemetry/mcp-add/output.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@

=== A new install for two agents: CHANGED per (service, agent); github needs sign-in, so its sign-in state is PENDING

>>> [CLI] aitools mcp add --profile logfood --names system.ai.genie_one_mcp,system.ai.github --agents claude-code,codex --output json
{
"calling_agent": "TYPE_UNSPECIFIED",
"operation": "ADD",
"servers": [
{
"service": "GENIE_ONE",
"agent": "CLAUDE_CODE",
"outcome": "CHANGED",
"sign_in_state": "NOT_REQUIRED"
},
{
"service": "GENIE_ONE",
"agent": "CODEX",
"outcome": "CHANGED",
"sign_in_state": "NOT_REQUIRED"
},
{
"service": "GITHUB",
"agent": "CLAUDE_CODE",
"outcome": "CHANGED",
"sign_in_state": "PENDING"
},
{
"service": "GITHUB",
"agent": "CODEX",
"outcome": "CHANGED",
"sign_in_state": "PENDING"
}
],
"bound_profile_count": 1
}

=== The same servers again are re-asserted: UNCHANGED

>>> [CLI] aitools mcp add --profile logfood --names system.ai.genie_one_mcp,system.ai.github --agents claude-code,codex --output json
{
"calling_agent": "TYPE_UNSPECIFIED",
"operation": "ADD",
"servers": [
{
"service": "GENIE_ONE",
"agent": "CLAUDE_CODE",
"outcome": "UNCHANGED",
"sign_in_state": "NOT_REQUIRED"
},
{
"service": "GENIE_ONE",
"agent": "CODEX",
"outcome": "UNCHANGED",
"sign_in_state": "NOT_REQUIRED"
},
{
"service": "GITHUB",
"agent": "CLAUDE_CODE",
"outcome": "UNCHANGED",
"sign_in_state": "PENDING"
},
{
"service": "GITHUB",
"agent": "CODEX",
"outcome": "UNCHANGED",
"sign_in_state": "PENDING"
}
],
"bound_profile_count": 1
}

=== An agent config that does not parse fails that agent only: FAILED with AGENT_CONFIG_INVALID

>>> musterr [CLI] aitools mcp add --profile logfood --names system.ai.genie_one_mcp --agents claude-code,codex --output json
{
"calling_agent": "TYPE_UNSPECIFIED",
"operation": "ADD",
"servers": [
{
"service": "GENIE_ONE",
"agent": "CLAUDE_CODE",
"outcome": "UNCHANGED",
"sign_in_state": "NOT_REQUIRED"
},
{
"service": "GENIE_ONE",
"agent": "CODEX",
"outcome": "FAILED",
"error_category": "AGENT_CONFIG_INVALID",
"sign_in_state": "NOT_REQUIRED"
}
],
"bound_profile_count": 1
}
22 changes: 22 additions & 0 deletions acceptance/cmd/aitools/telemetry/mcp-add/script
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
sethome home
mkdir -p home/.claude home/.codex
envsubst < .databrickscfg > out && mv out .databrickscfg
export DATABRICKS_CONFIG_FILE=.databrickscfg

# Seeding the fake is not an MCP command, so its uploads (if any) are dropped here.
MSYS_NO_PATHCONV=1 $CLI api post "/api/2.1/unity-catalog/mcp-services?parent=schemas/system.ai&mcp_service_id=genie_one_mcp" --profile logfood --json '{"comment":"Ask questions about your data"}' > /dev/null
MSYS_NO_PATHCONV=1 $CLI api post "/api/2.1/unity-catalog/mcp-services?parent=schemas/system.ai&mcp_service_id=github" --profile logfood --json '{"comment":"Repositories, issues, pull requests","config":{"source_connection":{"name":"connections/github","options":{"oauth_provider":"github"}}}}' > /dev/null
rm -f "$OUT_REQUESTS"

title "A new install for two agents: CHANGED per (service, agent); github needs sign-in, so its sign-in state is PENDING"
trace $CLI aitools mcp add --profile logfood --names system.ai.genie_one_mcp,system.ai.github --agents claude-code,codex --output json > /dev/null
mcp_events

title "The same servers again are re-asserted: UNCHANGED"
trace $CLI aitools mcp add --profile logfood --names system.ai.genie_one_mcp,system.ai.github --agents claude-code,codex --output json > /dev/null
mcp_events

title "An agent config that does not parse fails that agent only: FAILED with AGENT_CONFIG_INVALID"
cp codex-config.toml home/.codex/config.toml
trace musterr $CLI aitools mcp add --profile logfood --names system.ai.genie_one_mcp --agents claude-code,codex --output json > /dev/null
mcp_events
7 changes: 7 additions & 0 deletions acceptance/cmd/aitools/telemetry/mcp-remove/.databrickscfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[logfood]
host = $DATABRICKS_HOST
token = test-token-logfood

[dogfood]
host = $DATABRICKS_HOST
token = test-token-logfood
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Cloud = false
95 changes: 95 additions & 0 deletions acceptance/cmd/aitools/telemetry/mcp-remove/output.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@

=== Installed for two profiles: the bound profile count is 2 after the second add

>>> [CLI] aitools mcp add --profile logfood --names system.ai.genie_one_mcp --agents claude-code --output json
{
"calling_agent": "TYPE_UNSPECIFIED",
"operation": "ADD",
"servers": [
{
"service": "GENIE_ONE",
"agent": "CLAUDE_CODE",
"outcome": "CHANGED",
"sign_in_state": "NOT_REQUIRED"
}
],
"bound_profile_count": 1
}

>>> [CLI] aitools mcp add --profile dogfood --names system.ai.genie_one_mcp --agents claude-code --output json
{
"calling_agent": "TYPE_UNSPECIFIED",
"operation": "ADD",
"servers": [
{
"service": "GENIE_ONE",
"agent": "CLAUDE_CODE",
"outcome": "CHANGED",
"sign_in_state": "NOT_REQUIRED"
}
],
"bound_profile_count": 2
}

=== remove --profile dogfood: CHANGED for its agent entry, and one profile stays bound

>>> [CLI] aitools mcp remove --profile dogfood --output json
{
"calling_agent": "TYPE_UNSPECIFIED",
"operation": "REMOVE",
"servers": [
{
"service": "GENIE_ONE",
"agent": "CLAUDE_CODE",
"outcome": "CHANGED"
}
],
"bound_profile_count": 1
}

=== An entry changed outside this CLI is left in place: SKIPPED

>>> [CLI] aitools mcp remove --names genie_one_mcp-logfood --profile logfood --output json
{
"calling_agent": "TYPE_UNSPECIFIED",
"operation": "REMOVE",
"servers": [
{
"service": "GENIE_ONE",
"agent": "CLAUDE_CODE",
"outcome": "SKIPPED"
}
],
"bound_profile_count": 1
}

=== An entry already gone from the agent config is SKIPPED, and the state record goes

>>> [CLI] aitools mcp remove --names genie_one_mcp-logfood --profile logfood --output json
{
"calling_agent": "TYPE_UNSPECIFIED",
"operation": "REMOVE",
"servers": [
{
"service": "GENIE_ONE",
"agent": "CLAUDE_CODE",
"outcome": "CHANGED"
}
],
"bound_profile_count": 0
}

=== A record with no agent left goes with a REMOVE of its own: CHANGED with no agent, and no profile bound

>>> [CLI] aitools mcp remove --names dbsql-logfood --profile logfood --output json
{
"calling_agent": "TYPE_UNSPECIFIED",
"operation": "REMOVE",
"servers": [
{
"service": "DBSQL",
"outcome": "CHANGED"
}
],
"bound_profile_count": 0
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"schema_version": 3,
"mcp": {
"servers": [
{
"name": "dbsql-logfood",
"service": "system.ai.dbsql",
"profile": "logfood",
"host": "$DATABRICKS_HOST",
"workspace_id": "",
"sign_in_required": false,
"command": "",
"config_file": "$PWD/.databrickscfg",
"agents": {},
"installed_at": "2026-01-01T00:00:00Z",
"cli_version": "0.0.0"
}
]
}
}
33 changes: 33 additions & 0 deletions acceptance/cmd/aitools/telemetry/mcp-remove/script
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
sethome home
mkdir -p home/.claude
envsubst < .databrickscfg > out && mv out .databrickscfg
export DATABRICKS_CONFIG_FILE=.databrickscfg

MSYS_NO_PATHCONV=1 $CLI api post "/api/2.1/unity-catalog/mcp-services?parent=schemas/system.ai&mcp_service_id=genie_one_mcp" --profile logfood --json '{"comment":"Ask questions about your data"}' > /dev/null
rm -f "$OUT_REQUESTS"

title "Installed for two profiles: the bound profile count is 2 after the second add"
trace $CLI aitools mcp add --profile logfood --names system.ai.genie_one_mcp --agents claude-code --output json > /dev/null
mcp_events
trace $CLI aitools mcp add --profile dogfood --names system.ai.genie_one_mcp --agents claude-code --output json > /dev/null
mcp_events

title "remove --profile dogfood: CHANGED for its agent entry, and one profile stays bound"
trace $CLI aitools mcp remove --profile dogfood --output json > /dev/null
mcp_events

title "An entry changed outside this CLI is left in place: SKIPPED"
jq '.mcpServers["genie_one_mcp-logfood"].command = "/usr/bin/other"' home/.claude.json > home/claude.tmp && mv home/claude.tmp home/.claude.json
trace $CLI aitools mcp remove --names genie_one_mcp-logfood --profile logfood --output json > /dev/null
mcp_events

title "An entry already gone from the agent config is SKIPPED, and the state record goes"
jq 'del(.mcpServers["genie_one_mcp-logfood"])' home/.claude.json > home/claude.tmp && mv home/claude.tmp home/.claude.json
trace $CLI aitools mcp remove --names genie_one_mcp-logfood --profile logfood --output json > /dev/null
mcp_events

title "A record with no agent left goes with a REMOVE of its own: CHANGED with no agent, and no profile bound"
mkdir -p home/.databricks/aitools/skills
envsubst < record-only.state.json > home/.databricks/aitools/skills/.state.json
trace $CLI aitools mcp remove --names dbsql-logfood --profile logfood --output json > /dev/null
mcp_events
3 changes: 3 additions & 0 deletions acceptance/cmd/aitools/telemetry/mcp-serve/.databrickscfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[logfood]
host = $DATABRICKS_HOST
token = test-token-logfood
1 change: 1 addition & 0 deletions acceptance/cmd/aitools/telemetry/mcp-serve/out.test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Cloud = false
44 changes: 44 additions & 0 deletions acceptance/cmd/aitools/telemetry/mcp-serve/output.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@

=== serve logs PROXY_STARTED after the binding checks; the upload is sent when stdin closes

>>> [CLI] aitools mcp serve --profile logfood --service system.ai.genie_one_mcp --agent claude-code
No install record for system.ai.genie_one_mcp on profile "logfood"; using its current host [DATABRICKS_URL]
{
"calling_agent": "CLAUDE_CODE",
"operation": "PROXY_STARTED",
"servers": [
{
"service": "GENIE_ONE"
}
],
"bound_profile_count": 0
}

=== A binding that fails logs no PROXY_STARTED, so no upload follows

>>> musterr [CLI] aitools mcp serve --profile gone --service system.ai.genie_one_mcp --agent codex
MCP server system.ai.genie_one_mcp needs Databricks profile "gone", which does not exist in [TEST_TMP_DIR]/.databrickscfg. Set it up with: databricks auth login --profile gone

=== authenticate opens the sign-in page: PROXY_STARTED, then SIGN_IN_PROMPTED with the sign-in state after the wait
{
"calling_agent": "CLAUDE_CODE",
"operation": "PROXY_STARTED",
"servers": [
{
"service": "GITHUB"
}
],
"bound_profile_count": 0
}
{
"calling_agent": "CLAUDE_CODE",
"operation": "SIGN_IN_PROMPTED",
"servers": [
{
"service": "GITHUB",
"error_category": "SIGN_IN_REQUIRED",
"sign_in_state": "ACTIVE"
}
],
"bound_profile_count": 0
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"jsonrpc":"2.0","id":4,"method":"tools/list"}
{"jsonrpc":"2.0","id":5,"method":"tools/call","params":{"name":"echo","arguments":{"text":"hello"}}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"authenticate","arguments":{}}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"test-client","version":"0.0.1"}}}
{"jsonrpc":"2.0","method":"notifications/initialized"}
{"jsonrpc":"2.0","id":2,"method":"tools/list"}
32 changes: 32 additions & 0 deletions acceptance/cmd/aitools/telemetry/mcp-serve/script
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
sethome home
envsubst < .databrickscfg > out && mv out .databrickscfg
export DATABRICKS_CONFIG_FILE=.databrickscfg

# No browser: the proxy prints the sign-in URL and does not open anything.
export BROWSER=none
export RETRY_MAX_ATTEMPTS=60
# Seeding the fake is not an MCP command, so its uploads (if any) are dropped here.
MSYS_NO_PATHCONV=1 $CLI api post "/api/2.1/unity-catalog/mcp-services?parent=schemas/system.ai&mcp_service_id=genie_one_mcp" --profile logfood --json '{"comment":"Ask questions about your data"}' > /dev/null
MSYS_NO_PATHCONV=1 $CLI api post "/api/2.1/unity-catalog/mcp-services?parent=schemas/system.ai&mcp_service_id=github" --profile logfood --json '{"comment":"Repositories, issues, pull requests","config":{"source_connection":{"name":"connections/github","options":{"oauth_provider":"github"}}}}' > /dev/null
rm -f "$OUT_REQUESTS"

title "serve logs PROXY_STARTED after the binding checks; the upload is sent when stdin closes"
trace $CLI aitools mcp serve --profile logfood --service system.ai.genie_one_mcp --agent claude-code < requests-list.ndjson > /dev/null
mcp_events

title "A binding that fails logs no PROXY_STARTED, so no upload follows"
trace musterr $CLI aitools mcp serve --profile gone --service system.ai.genie_one_mcp --agent codex < /dev/null > /dev/null
mcp_events

title "authenticate opens the sign-in page: PROXY_STARTED, then SIGN_IN_PROMPTED with the sign-in state after the wait"
{
cat requests-list.ndjson
retry --until '"id":2' cat serve.stdout.txt > /dev/null
cat requests-authenticate.ndjson
retry --until "Waiting for sign-in" cat serve.stderr.txt > /dev/null
MSYS_NO_PATHCONV=1 $CLI api post /api/2.1/unity-catalog/mcp-services/system.ai.github/user-credentials --profile logfood --json '{}' > /dev/null
retry --until '"id":3' cat serve.stdout.txt > /dev/null
cat requests-after.ndjson
} | $CLI aitools mcp serve --profile logfood --service system.ai.github --agent claude-code > serve.stdout.txt 2> serve.stderr.txt
mcp_events
rm serve.stdout.txt serve.stderr.txt
3 changes: 3 additions & 0 deletions acceptance/cmd/aitools/telemetry/mcp-sign-in/.databrickscfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[logfood]
host = $DATABRICKS_HOST
token = test-token-logfood
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Cloud = false
Loading
Loading