Skip to content

Record the shown error code and category on command telemetry - #517

Draft
joe4dev wants to merge 7 commits into
devx-1004-distinguish-proxied-tool-exits-from-lstk-errors-in-commandfrom
devx-1004-error-code-telemetry
Draft

joe4dev wants to merge 7 commits into
devx-1004-distinguish-proxied-tool-exits-from-lstk-errors-in-commandfrom
devx-1004-error-code-telemetry

Conversation

@joe4dev

@joe4dev joe4dev commented Sep 25, 2026 •

Copy link
Copy Markdown
Member

Motivation

With proxied, proxy_exit_code and cancelled (#499), telemetry can tell whose failure a failed lstk_command was, but not why: lstk aws --account 123 s3 ls, aws missing from PATH, Docker being down and a genuine lstk defect all land in the same "lstk failed" bucket, which is why that bucket cannot be read as a reliability metric. The repo already has the why axis, output.ErrorCode with its categories, but it rode the ErrorEvent the sink consumed, not the error value the telemetry hook receives, and only 20 of 67 error sites set one.

Solution

Stacked on #499. lstk_command events gain result.error_code and result.error_category, present when the failing site classified its error, absent otherwise.

  • Carrier. SilentError gains a Code, set by output.Fail(sink, event, err), which emits the event and returns the silent error in one call, replacing the two-step sink.Emit(...); return output.NewSilentError(err) that dropped the code. Domain code that returns errors for the caller to render uses output.WithCode(err, code), a non-silent carrier. commandResult reads either via output.ErrorCodeOf and emits the code and its category.
  • Classification. Every ErrorEvent in the tree now sets a Code. emitValidationError is split by cause (VALIDATION_ERROR for arguments, NETWORK_ERROR for an unreachable --endpoint-url, EMULATOR_WRONG_TYPE for a wrong emulator behind it); the fifteen runtime not healthy preflights share runtime.UnhealthyError; terraform backend provisioning is IAC_DEPLOY_FAILED; Azure setup and interception failures are EMULATOR_NOT_RUNNING, INTERNAL_ERROR or VALIDATION_ERROR; snapshot, auth, config, dependency and container sites got their matching codes. Three definitions were widened: DEPENDENCY_MISSING (also an unusable tool version), SNAPSHOT_INVALID_REF (also an invalid or incompatible snapshot file), EMULATOR_START_FAILED (also a container-name conflict or unremovable leftover container).
  • Ratchet. TestEveryNewErrorEventSetsACode parses cmd/ and internal/ and fails when an ErrorEvent literal has no Code. Failures that reach Cobra's fallback printer without any event stay unclassified; the pipe reports that share.

Not emitted: CANCELLED (the cancelled field is the observation) and anything for proxied tool exits (no ErrorEvent is shown for them; proxy_exit_code is their axis).

Analytics side

The reliability and coverage queries, and the category split (USAGE, CONFIG, AUTH, RUNTIME are the user's environment; the rest is a team decision), are in openspec/changes/distinguish-proxied-command-errors/design.md, Decision 8.

Testing

Unit tests for the carriers and the builder; integration tests for VALIDATION_ERROR, RUNTIME_UNAVAILABLE, NETWORK_ERROR, EMULATOR_WRONG_TYPE, INTERNAL_ERROR, EMULATOR_NOT_RUNNING, IAC_DEPLOY_FAILED, INTEGRATION_NOT_SET_UP, AUTH_REQUIRED and SNAPSHOT_NOT_FOUND. The 40-scenario manual matrix on #499 was run with these commits and every classified row matched.

Docs

Every ErrorEvent now carries an output.ErrorCode, so --json envelopes that previously fell back to INTERNAL_ERROR report their real code, and three code definitions in docs/structured-output.md were widened (see above). The telemetry payload itself is internal and undocumented.

Review

Human review advised: it touches 46 files across most packages, changes documented --json error codes, and makes taxonomy calls (INTERNAL_ERROR for lstk's own failing subprocesses, IAC_FILE_NOT_FOUND for a missing provider) that deserve a second opinion.

Todo

🤖 Generated with Claude Code

Follow-up of DEVX-1004

@joe4dev joe4dev added semver: patch docs: needed Pull request requires documentation updates labels Sep 25, 2026
joe4dev and others added 7 commits September 25, 2026 09:55
…metry

Co-Authored-By: Claude <noreply@anthropic.com>
…etry

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
@joe4dev
joe4dev force-pushed the devx-1004-error-code-telemetry branch from 97e23b2 to 84b933e Compare September 25, 2026 07:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs: needed Pull request requires documentation updates semver: patch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant