Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
96bcd1d
Add PLAN.md for bundle bitmap
denik Jul 29, 2026
f4dc05f
Add bundle bitmap for per-field telemetry
denik Jul 29, 2026
2692a9a
Remove PLAN.md
denik Jul 29, 2026
2c3d106
Add bool value bits, decode round-trip, and template recording
denik Jul 29, 2026
fabebfd
Move boolean telemetry into a typed Telemetry struct
denik Jul 30, 2026
8debb3f
Regenerate goldens after rebase; ignore .databricks in bitmap test
denik Jul 30, 2026
f689310
Fix bitmap test: executable decode script and deterministic cache
denik Jul 30, 2026
eecf670
Strip CR when parsing embedded bitmap schema
denik Jul 30, 2026
3291975
Regenerate bitmap schema from scratch; refresh goldens in generate task
denik Jul 30, 2026
3f7fbef
Make Telemetry additive: mirror bool metrics into a struct on Bundle
denik Jul 31, 2026
c715983
Record DMS undeclared-writer flags as True/False pairs in bitmap
denik Jul 31, 2026
d790455
Revert DMS flags to single bits: false is not lost here
denik Aug 3, 2026
b1e7d71
Fix bitmap encoding every telemetry bit as set
denik Aug 3, 2026
0ee45e6
Regenerate bitmap schema and drop removed Local test option
denik Aug 11, 2026
86f051d
Regenerate bitmap schema and goldens after rebase
denik Aug 17, 2026
c09fc2e
Port bitmap telemetry mirror to current main
denik Nov 3, 2026
07caa29
Regenerate bitmap schema after rebase (data_quality_monitors)
denik Nov 3, 2026
25cce61
Bitmap: record user-authored config; fix presence detection; simplify…
denik Nov 4, 2026
aba0180
Add a dedicated bitmap test that records every schema line
denik Nov 4, 2026
7b7fa25
Make Telemetry the source of bool_values; encode bool values; add decode
denik Nov 4, 2026
24b6fae
Add a self-contained bitmap decoder to databricks-bundles
denik Nov 4, 2026
a3a8602
Regenerate bitmap schema after rebase (variables.*.sensitive)
denik Nov 4, 2026
864a8ba
Regenerate bitmap schema after rebase (materialization_sets)
denik Nov 4, 2026
2682853
Keep the bitmap decoder out of pydabs codegen cleanup and docs
denik Nov 4, 2026
b283061
no_drift: log bitmap stderr instead of diffing it
denik Nov 4, 2026
684ebea
Add a local-only invariant test for validate and the bitmap round trip
denik Nov 4, 2026
93082b5
Update selftest/selection for the new validate_bitmap invariant test
denik Nov 4, 2026
5d03d6f
bitmap: one bitmap command with -o compressed|text|json
denik Nov 5, 2026
64220f4
bitmap/full: split the schema golden per resource type
denik Nov 5, 2026
888851f
bitmap/presence: lifecycle.started true on one cluster and false on a…
denik Nov 5, 2026
cb34fa1
Report each lifecycle.started value seen instead of a random resource's
denik Nov 5, 2026
6eddf47
Regenerate bitmap schema after rebase (postgres_data_apis)
denik Nov 5, 2026
6c2229f
Telemetry: BoolSet.SetBool instead of appending to []bool
denik Nov 6, 2026
401b0ef
bitmap: compute bits early and fill in variable references later
denik Nov 9, 2026
0563448
Adapt to optional config bools being *bool
denik Nov 9, 2026
9c777e4
Regenerate bitmap schema after rebase (feature_specs, materialization…
denik Nov 11, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/rules/telemetry.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ The deploy payload is declared twice: the proto in the universe repo (`proto/log

Other things worth knowing:

- A one-off boolean needs no proto change: add a key constant to `bundle/metrics/metrics.go` (snake_case, matching the constant name) and call `b.Metrics.SetBoolValue(...)`. It lands in `experimental.bool_values`.
- A one-off boolean needs no proto change: add a field to `bundle.Telemetry` (`bundle/telemetry.go`) with a snake_case `metric:"..."` tag and set it. Pick the type by what is measured: `bool` for a flag reported only when true, `*bool` for a measured value (nil when not measured), `BoolSet` for a measurement taken several times (e.g. once per artifact; record each with `SetBool`, and each value taken is reported once). It lands in `experimental.bool_values` and in the bundle bitmap; run `./task generate-bitmap-schema` to append it to the bitmap schema.
- Metrics that may be removed later belong in `BundleDeployExperimental`, which carries no compatibility guarantees.
- Positional encodings such as the `upload_file_sizes` histogram are frozen once the metric has adoption: changing a bucket bound silently changes the meaning of every previously reported entry.
- Per-resource state sizes come from the direct engine's state in `resources.json`.
Expand Down
29 changes: 29 additions & 0 deletions Taskfile.yml
Original file line number Diff line number Diff line change
Expand Up @@ -808,6 +808,7 @@ tasks:
# Refreshes acceptance/bundle/refschema/output/*.txt, which feeds
# generate-direct-apitypes and generate-direct-resources below.
- task: generate-refschema
- task: generate-bitmap-schema
- task: generate-schema
- task: generate-validation
- task: generate-direct
Expand All @@ -826,6 +827,7 @@ tasks:
cmds:
- task: generate-cligen
- task: generate-refschema
- task: generate-bitmap-schema
- task: generate-schema
- task: generate-validation
- task: generate-direct
Expand Down Expand Up @@ -1002,6 +1004,33 @@ tasks:
cmds:
- go test ./acceptance -run TestAccept/bundle/refschema -update

# Appends any newly added bundle config fields to bundle/bitmap/schema.txt, the
# embedded bitmap schema (see cmd/bundle/bitmap). The schema is append-only, so
# update-schema reads the currently embedded schema and only appends. Output is
# written to a temp file first so the redirect does not truncate schema.txt
# before `go run` embeds it at compile time. Also refreshes the schema embedded
# in the databricks-bundles decoder (python/databricks/bundles/_bitmap) and the
# listing in acceptance/bundle/bitmap/full, which tracks every schema line.
generate-bitmap-schema:
desc: Regenerate bundle/bitmap/schema.txt and its acceptance listing
sources:
- bundle/**/*.go
- exclude: bundle/**/*_test.go
- acceptance/bundle/bitmap/full/script
- acceptance/bundle/bitmap/full/split.py
- go.mod
- go.sum
- "{{.EMBED_SOURCES}}"
generates:
- bundle/bitmap/schema.txt
- acceptance/bundle/bitmap/full/output/*.txt
- python/databricks/bundles/_bitmap/_schema.py
cmds:
- go run . bundle bitmap update-schema > bundle/bitmap/schema.txt.tmp
- mv bundle/bitmap/schema.txt.tmp bundle/bitmap/schema.txt
- go run ./bundle/bitmap/genpython python/databricks/bundles/_bitmap/_schema.py
- go test ./acceptance -run TestAccept/bundle/bitmap/full -update

# Upstream field documentation comes from the checked-in .codegen/cli.json;
# bundle/internal/schema/annotations.yml carries the CLI-owned docs and
# overrides and is rewritten in place (synced with the config structure).
Expand Down
2 changes: 2 additions & 0 deletions acceptance/bundle/bitmap/full/out.test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Cloud = false
EnvMatrix.DMS = []
Empty file.
75 changes: 75 additions & 0 deletions acceptance/bundle/bitmap/full/output/alerts.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
resources.alerts 4228
resources.alerts.* 4229
resources.alerts.*.url 4230
resources.alerts.*.modified_status 4231
resources.alerts.*.lifecycle 4232
resources.alerts.*.lifecycle.prevent_destroy 4233
resources.alerts.*.lifecycle.prevent_destroy=true 4234
resources.alerts.*.lifecycle.prevent_destroy=false 4235
resources.alerts.*.lifecycle.depends_on 4236
resources.alerts.*.lifecycle.depends_on[*] 4237
resources.alerts.*.lifecycle.depends_on[*].resource 4238
resources.alerts.*.create_time 4239
resources.alerts.*.custom_description 4240
resources.alerts.*.custom_summary 4241
resources.alerts.*.custom_template_format 4242
resources.alerts.*.display_name 4243
resources.alerts.*.effective_run_as 4244
resources.alerts.*.effective_run_as.service_principal_name 4245
resources.alerts.*.effective_run_as.user_name 4246
resources.alerts.*.evaluation 4247
resources.alerts.*.evaluation.comparison_operator 4248
resources.alerts.*.evaluation.empty_result_state 4249
resources.alerts.*.evaluation.last_evaluated_at 4250
resources.alerts.*.evaluation.notification 4251
resources.alerts.*.evaluation.notification.notify_on_ok 4252
resources.alerts.*.evaluation.notification.notify_on_ok=true 4253
resources.alerts.*.evaluation.notification.notify_on_ok=false 4254
resources.alerts.*.evaluation.notification.retrigger_seconds 4255
resources.alerts.*.evaluation.notification.subscriptions 4256
resources.alerts.*.evaluation.notification.subscriptions[*] 4257
resources.alerts.*.evaluation.notification.subscriptions[*].destination_id 4258
resources.alerts.*.evaluation.notification.subscriptions[*].user_email 4259
resources.alerts.*.evaluation.source 4260
resources.alerts.*.evaluation.source.aggregation 4261
resources.alerts.*.evaluation.source.display 4262
resources.alerts.*.evaluation.source.name 4263
resources.alerts.*.evaluation.state 4264
resources.alerts.*.evaluation.threshold 4265
resources.alerts.*.evaluation.threshold.column 4266
resources.alerts.*.evaluation.threshold.column.aggregation 4267
resources.alerts.*.evaluation.threshold.column.display 4268
resources.alerts.*.evaluation.threshold.column.name 4269
resources.alerts.*.evaluation.threshold.value 4270
resources.alerts.*.evaluation.threshold.value.bool_value 4271
resources.alerts.*.evaluation.threshold.value.bool_value=true 4272
resources.alerts.*.evaluation.threshold.value.bool_value=false 4273
resources.alerts.*.evaluation.threshold.value.double_value 4274
resources.alerts.*.evaluation.threshold.value.string_value 4275
resources.alerts.*.id 4276
resources.alerts.*.lifecycle_state 4277
resources.alerts.*.owner_user_name 4278
resources.alerts.*.parameters 4279
resources.alerts.*.parameters[*] 4280
resources.alerts.*.parameters[*].name 4281
resources.alerts.*.parameters[*].type 4282
resources.alerts.*.parameters[*].value 4283
resources.alerts.*.parent_path 4284
resources.alerts.*.query_text 4285
resources.alerts.*.run_as 4286
resources.alerts.*.run_as.service_principal_name 4287
resources.alerts.*.run_as.user_name 4288
resources.alerts.*.run_as_user_name 4289
resources.alerts.*.schedule 4290
resources.alerts.*.schedule.pause_status 4291
resources.alerts.*.schedule.quartz_cron_schedule 4292
resources.alerts.*.schedule.timezone_id 4293
resources.alerts.*.update_time 4294
resources.alerts.*.warehouse_id 4295
resources.alerts.*.permissions 4296
resources.alerts.*.permissions[*] 4297
resources.alerts.*.permissions[*].level 4298
resources.alerts.*.permissions[*].user_name 4299
resources.alerts.*.permissions[*].service_principal_name 4300
resources.alerts.*.permissions[*].group_name 4301
resources.alerts.*.file_path 4302
215 changes: 215 additions & 0 deletions acceptance/bundle/bitmap/full/output/apps.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,215 @@
resources.apps 3990
resources.apps.* 3991
resources.apps.*.url 3992
resources.apps.*.modified_status 3993
resources.apps.*.lifecycle 3994
resources.apps.*.lifecycle.prevent_destroy 3995
resources.apps.*.lifecycle.prevent_destroy=true 3996
resources.apps.*.lifecycle.prevent_destroy=false 3997
resources.apps.*.lifecycle.depends_on 3998
resources.apps.*.lifecycle.depends_on[*] 3999
resources.apps.*.lifecycle.depends_on[*].resource 4000
resources.apps.*.active_deployment 4001
resources.apps.*.active_deployment.command 4002
resources.apps.*.active_deployment.command[*] 4003
resources.apps.*.active_deployment.create_time 4004
resources.apps.*.active_deployment.creator 4005
resources.apps.*.active_deployment.deployment_artifacts 4006
resources.apps.*.active_deployment.deployment_artifacts.source_code_path 4007
resources.apps.*.active_deployment.deployment_id 4008
resources.apps.*.active_deployment.env_vars 4009
resources.apps.*.active_deployment.env_vars[*] 4010
resources.apps.*.active_deployment.env_vars[*].name 4011
resources.apps.*.active_deployment.env_vars[*].value 4012
resources.apps.*.active_deployment.env_vars[*].value_from 4013
resources.apps.*.active_deployment.git_source 4014
resources.apps.*.active_deployment.git_source.branch 4015
resources.apps.*.active_deployment.git_source.commit 4016
resources.apps.*.active_deployment.git_source.git_repository 4017
resources.apps.*.active_deployment.git_source.git_repository.auto_deploy 4018
resources.apps.*.active_deployment.git_source.git_repository.auto_deploy=true 4019
resources.apps.*.active_deployment.git_source.git_repository.auto_deploy=false 4020
resources.apps.*.active_deployment.git_source.git_repository.caller_credential_id 4021
resources.apps.*.active_deployment.git_source.git_repository.provider 4022
resources.apps.*.active_deployment.git_source.git_repository.url 4023
resources.apps.*.active_deployment.git_source.resolved_commit 4024
resources.apps.*.active_deployment.git_source.source_code_path 4025
resources.apps.*.active_deployment.git_source.tag 4026
resources.apps.*.active_deployment.health_check 4027
resources.apps.*.active_deployment.health_check.path 4028
resources.apps.*.active_deployment.health_check.timeout 4029
resources.apps.*.active_deployment.mode 4030
resources.apps.*.active_deployment.source_code_path 4031
resources.apps.*.active_deployment.status 4032
resources.apps.*.active_deployment.status.message 4033
resources.apps.*.active_deployment.status.state 4034
resources.apps.*.active_deployment.update_time 4035
resources.apps.*.app_status 4036
resources.apps.*.app_status.message 4037
resources.apps.*.app_status.running_instances 4038
resources.apps.*.app_status.state 4039
resources.apps.*.budget_policy_id 4040
resources.apps.*.compute_max_instances 4041
resources.apps.*.compute_min_instances 4042
resources.apps.*.compute_size 4043
resources.apps.*.compute_status 4044
resources.apps.*.compute_status.active_instances 4045
resources.apps.*.compute_status.message 4046
resources.apps.*.compute_status.state 4047
resources.apps.*.create_time 4048
resources.apps.*.creator 4049
resources.apps.*.default_git_source 4050
resources.apps.*.default_git_source.branch 4051
resources.apps.*.default_git_source.commit 4052
resources.apps.*.default_git_source.git_repository 4053
resources.apps.*.default_git_source.git_repository.auto_deploy 4054
resources.apps.*.default_git_source.git_repository.auto_deploy=true 4055
resources.apps.*.default_git_source.git_repository.auto_deploy=false 4056
resources.apps.*.default_git_source.git_repository.caller_credential_id 4057
resources.apps.*.default_git_source.git_repository.provider 4058
resources.apps.*.default_git_source.git_repository.url 4059
resources.apps.*.default_git_source.resolved_commit 4060
resources.apps.*.default_git_source.source_code_path 4061
resources.apps.*.default_git_source.tag 4062
resources.apps.*.default_source_code_path 4063
resources.apps.*.description 4064
resources.apps.*.effective_budget_policy_id 4065
resources.apps.*.effective_usage_policy_id 4066
resources.apps.*.effective_user_api_scopes 4067
resources.apps.*.effective_user_api_scopes[*] 4068
resources.apps.*.forward_user_access_token 4069
resources.apps.*.forward_user_access_token=true 4070
resources.apps.*.forward_user_access_token=false 4071
resources.apps.*.git_repository 4072
resources.apps.*.git_repository.auto_deploy 4073
resources.apps.*.git_repository.auto_deploy=true 4074
resources.apps.*.git_repository.auto_deploy=false 4075
resources.apps.*.git_repository.caller_credential_id 4076
resources.apps.*.git_repository.provider 4077
resources.apps.*.git_repository.url 4078
resources.apps.*.git_source 4079
resources.apps.*.git_source.branch 4080
resources.apps.*.git_source.commit 4081
resources.apps.*.git_source.git_repository 4082
resources.apps.*.git_source.git_repository.auto_deploy 4083
resources.apps.*.git_source.git_repository.auto_deploy=true 4084
resources.apps.*.git_source.git_repository.auto_deploy=false 4085
resources.apps.*.git_source.git_repository.caller_credential_id 4086
resources.apps.*.git_source.git_repository.provider 4087
resources.apps.*.git_source.git_repository.url 4088
resources.apps.*.git_source.resolved_commit 4089
resources.apps.*.git_source.source_code_path 4090
resources.apps.*.git_source.tag 4091
resources.apps.*.id 4092
resources.apps.*.name 4093
resources.apps.*.oauth2_app_client_id 4094
resources.apps.*.oauth2_app_integration_id 4095
resources.apps.*.pending_deployment 4096
resources.apps.*.pending_deployment.command 4097
resources.apps.*.pending_deployment.command[*] 4098
resources.apps.*.pending_deployment.create_time 4099
resources.apps.*.pending_deployment.creator 4100
resources.apps.*.pending_deployment.deployment_artifacts 4101
resources.apps.*.pending_deployment.deployment_artifacts.source_code_path 4102
resources.apps.*.pending_deployment.deployment_id 4103
resources.apps.*.pending_deployment.env_vars 4104
resources.apps.*.pending_deployment.env_vars[*] 4105
resources.apps.*.pending_deployment.env_vars[*].name 4106
resources.apps.*.pending_deployment.env_vars[*].value 4107
resources.apps.*.pending_deployment.env_vars[*].value_from 4108
resources.apps.*.pending_deployment.git_source 4109
resources.apps.*.pending_deployment.git_source.branch 4110
resources.apps.*.pending_deployment.git_source.commit 4111
resources.apps.*.pending_deployment.git_source.git_repository 4112
resources.apps.*.pending_deployment.git_source.git_repository.auto_deploy 4113
resources.apps.*.pending_deployment.git_source.git_repository.auto_deploy=true 4114
resources.apps.*.pending_deployment.git_source.git_repository.auto_deploy=false 4115
resources.apps.*.pending_deployment.git_source.git_repository.caller_credential_id 4116
resources.apps.*.pending_deployment.git_source.git_repository.provider 4117
resources.apps.*.pending_deployment.git_source.git_repository.url 4118
resources.apps.*.pending_deployment.git_source.resolved_commit 4119
resources.apps.*.pending_deployment.git_source.source_code_path 4120
resources.apps.*.pending_deployment.git_source.tag 4121
resources.apps.*.pending_deployment.health_check 4122
resources.apps.*.pending_deployment.health_check.path 4123
resources.apps.*.pending_deployment.health_check.timeout 4124
resources.apps.*.pending_deployment.mode 4125
resources.apps.*.pending_deployment.source_code_path 4126
resources.apps.*.pending_deployment.status 4127
resources.apps.*.pending_deployment.status.message 4128
resources.apps.*.pending_deployment.status.state 4129
resources.apps.*.pending_deployment.update_time 4130
resources.apps.*.resources 4131
resources.apps.*.resources[*] 4132
resources.apps.*.resources[*].app 4133
resources.apps.*.resources[*].app.name 4134
resources.apps.*.resources[*].app.permission 4135
resources.apps.*.resources[*].database 4136
resources.apps.*.resources[*].database.database_name 4137
resources.apps.*.resources[*].database.instance_name 4138
resources.apps.*.resources[*].database.permission 4139
resources.apps.*.resources[*].description 4140
resources.apps.*.resources[*].experiment 4141
resources.apps.*.resources[*].experiment.experiment_id 4142
resources.apps.*.resources[*].experiment.permission 4143
resources.apps.*.resources[*].genie_space 4144
resources.apps.*.resources[*].genie_space.name 4145
resources.apps.*.resources[*].genie_space.permission 4146
resources.apps.*.resources[*].genie_space.space_id 4147
resources.apps.*.resources[*].job 4148
resources.apps.*.resources[*].job.id 4149
resources.apps.*.resources[*].job.permission 4150
resources.apps.*.resources[*].name 4151
resources.apps.*.resources[*].postgres 4152
resources.apps.*.resources[*].postgres.branch 4153
resources.apps.*.resources[*].postgres.database 4154
resources.apps.*.resources[*].postgres.permission 4155
resources.apps.*.resources[*].secret 4156
resources.apps.*.resources[*].secret.key 4157
resources.apps.*.resources[*].secret.permission 4158
resources.apps.*.resources[*].secret.scope 4159
resources.apps.*.resources[*].serving_endpoint 4160
resources.apps.*.resources[*].serving_endpoint.name 4161
resources.apps.*.resources[*].serving_endpoint.permission 4162
resources.apps.*.resources[*].sql_warehouse 4163
resources.apps.*.resources[*].sql_warehouse.id 4164
resources.apps.*.resources[*].sql_warehouse.permission 4165
resources.apps.*.resources[*].uc_securable 4166
resources.apps.*.resources[*].uc_securable.permission 4167
resources.apps.*.resources[*].uc_securable.securable_full_name 4168
resources.apps.*.resources[*].uc_securable.securable_kind 4169
resources.apps.*.resources[*].uc_securable.securable_type 4170
resources.apps.*.service_principal_client_id 4171
resources.apps.*.service_principal_id 4172
resources.apps.*.service_principal_name 4173
resources.apps.*.source_code_path 4174
resources.apps.*.space 4175
resources.apps.*.telemetry_export_destinations 4176
resources.apps.*.telemetry_export_destinations[*] 4177
resources.apps.*.telemetry_export_destinations[*].unity_catalog 4178
resources.apps.*.telemetry_export_destinations[*].unity_catalog.logs_table 4179
resources.apps.*.telemetry_export_destinations[*].unity_catalog.metrics_table 4180
resources.apps.*.telemetry_export_destinations[*].unity_catalog.traces_table 4181
resources.apps.*.thumbnail_url 4182
resources.apps.*.update_time 4183
resources.apps.*.updater 4184
resources.apps.*.usage_policy_id 4185
resources.apps.*.user_api_scopes 4186
resources.apps.*.user_api_scopes[*] 4187
resources.apps.*.lifecycle.started 4188
resources.apps.*.lifecycle.started=true 4189
resources.apps.*.lifecycle.started=false 4190
resources.apps.*.config 4191
resources.apps.*.config.command 4192
resources.apps.*.config.command[*] 4193
resources.apps.*.config.env 4194
resources.apps.*.config.env[*] 4195
resources.apps.*.config.env[*].name 4196
resources.apps.*.config.env[*].value 4197
resources.apps.*.config.env[*].value_from 4198
resources.apps.*.permissions 4199
resources.apps.*.permissions[*] 4200
resources.apps.*.permissions[*].level 4201
resources.apps.*.permissions[*].user_name 4202
resources.apps.*.permissions[*].service_principal_name 4203
resources.apps.*.permissions[*].group_name 4204
Loading
Loading