Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
4 changes: 4 additions & 0 deletions acceptance/air/cancel/output.txt
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@

=== cancel by id (text)

>>> [CLI] air cancel 123
Successfully requested cancellation for run 123

=== cancel by id (json)

>>> [CLI] air cancel 123 -o json
{
"v": 1,
Expand All @@ -16,12 +18,14 @@ Successfully requested cancellation for run 123
}

=== cancel multiple ids

>>> [CLI] air cancel 123 456
Successfully requested cancellation for run 123
Successfully requested cancellation for run 456
Successfully requested cancellation for 2 run(s).

=== cancel --all

>>> [CLI] air cancel --all -y
Searching active runs for [USERNAME] in [DATABRICKS_URL]...
Successfully requested cancellation for run 334747067049496
Expand Down
11 changes: 11 additions & 0 deletions acceptance/air/config-help/output.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@

=== command help

>>> [CLI] air run --help
Submit a training workload to Databricks serverless GPU compute.

Expand Down Expand Up @@ -32,6 +33,7 @@ Global Flags:
-t, --target string bundle target to use (if applicable)

=== schema overview

>>> [CLI] air run -h config
config
The run YAML schema. Pass a field path for details, e.g. config.compute.accelerator_type.
Expand Down Expand Up @@ -70,6 +72,7 @@ config
Use "-h config.<field>" for details on a field.

=== nested object lists its fields

>>> [CLI] air run -h config.compute
config.compute
Which accelerators to run on and how many.
Expand All @@ -88,6 +91,7 @@ config.compute
Use "-h config.compute.<field>" for details on a field.

=== new submission fields are documented

>>> [CLI] air run -h config.mlflow_artifact_location
config.mlflow_artifact_location
DBFS location where MLflow artifacts are written. A /Volumes path is normalized to dbfs:/Volumes/... .
Expand Down Expand Up @@ -124,6 +128,7 @@ config.environment.version
Required: no

=== leaf field

>>> [CLI] air run -h config.compute.accelerator_type
config.compute.accelerator_type
Which accelerator to run on, e.g. GPU_1xA10. See https://docs.databricks.com/aws/en/machine-learning/ai-runtime/cli/yaml-config#reference for the current list of supported GPU types. Matched case-sensitively.
Expand All @@ -137,6 +142,7 @@ config.compute.accelerator_type
Required: no

=== code source fields

>>> [CLI] air run -h config.code_source
config.code_source
Local code to upload and make available to the run.
Expand All @@ -150,6 +156,7 @@ config.code_source
Use "-h config.code_source.<field>" for details on a field.

=== union field

>>> [CLI] air run -h config.code_source.git.remote
config.code_source.git.remote
No longer supported: the snapshot archives your local copy. Only false is accepted; use commit to pin a revision.
Expand All @@ -158,6 +165,7 @@ config.code_source.git.remote
Required: no

=== git-pinned subdirectory documents root_path scoping

>>> [CLI] air run -h config.code_source.root_path
config.code_source.root_path
Root of the code source to archive. A git-pinned subdirectory packages only that subtree.
Expand All @@ -166,6 +174,7 @@ config.code_source.root_path
Required: when code_source is set

=== the config. prefix is optional

>>> [CLI] air run -h compute.num_accelerators
config.compute.num_accelerators
Total number of GPUs to allocate. Must be a positive multiple of the accelerator type's per-node GPU count. See https://docs.databricks.com/aws/en/machine-learning/ai-runtime/cli/yaml-config#reference for supported GPU types.
Expand All @@ -174,11 +183,13 @@ config.compute.num_accelerators
Required: no

=== unknown field suggests the near match

>>> [CLI] air run -h config.compute.acclerator_type
Error: unknown config field "config.compute.acclerator_type"; did you mean "accelerator_type"?

fields under "config.compute" are: accelerator_type, num_accelerators, pool_id, priority_class

=== free-form map keys are not schema fields

>>> [CLI] air run -h config.parameters.learning_rate
Error: "config.parameters" holds user-defined keys, so "learning_rate" is not part of the schema; see "config.parameters" instead
11 changes: 11 additions & 0 deletions acceptance/air/convert-to-dabs/output.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@

=== convert an AIR run YAML into a DABs bundle (in place, next to the source)

>>> [CLI] air convert-to-dabs train.yaml
Wrote a Databricks Asset Bundle to .:
databricks.yml
Expand All @@ -20,6 +21,7 @@ job and its uploaded files with:
[CLI] bundle destroy

=== emitted databricks.yml (code_source packaged as a tgz artifact; no code is copied)

>>> cat databricks.yml
bundle:
name: torchrun-a10-smoke-test
Expand Down Expand Up @@ -65,11 +67,13 @@ resources:
environment_version: "6"

=== the generated command.sh carries the run command

>>> cat generated_artifacts/command.sh
cd $CODE_SOURCE_PATH
torchrun --nproc_per_node=1 train.py

=== the emitted bundle validates

>>> [CLI] bundle validate
Name: torchrun-a10-smoke-test
Target: dev
Expand All @@ -80,12 +84,14 @@ Workspace:
Validation OK!

=== re-converting refuses to clobber the generated bundle

>>> [CLI] air convert-to-dabs train.yaml
Error: databricks.yml already exists in .; pass --force to overwrite or remove it

Exit code: 1

=== --force overwrites it

>>> [CLI] air convert-to-dabs train.yaml --force
Wrote a Databricks Asset Bundle to .:
databricks.yml
Expand All @@ -106,6 +112,7 @@ job and its uploaded files with:
[CLI] bundle destroy

=== convert a B300 run using the current SDK

>>> [CLI] air convert-to-dabs b300.yaml --output-dir generated-b300
Wrote a Databricks Asset Bundle to generated-b300:
databricks.yml
Expand All @@ -127,6 +134,7 @@ job and its uploaded files with:
[CLI] bundle destroy

=== the B300 bundle preserves its accelerator and AI Runtime version

>>> cat generated-b300/databricks.yml
bundle:
name: b300-smoke-test
Expand Down Expand Up @@ -159,6 +167,7 @@ resources:
environment_version: "6"

=== the current SDK recognizes B300 so bundle validation is clean

>>> [CLI] bundle validate
Name: b300-smoke-test
Target: dev
Expand All @@ -169,6 +178,7 @@ Workspace:
Validation OK!

=== databricks_ai versions use a base environment

>>> [CLI] air convert-to-dabs databricks-ai.yaml --force
Wrote a Databricks Asset Bundle to .:
databricks.yml
Expand All @@ -192,6 +202,7 @@ job and its uploaded files with:
base_environment: workspace-base-environments/databricks_ai_v5

=== the Databricks AI bundle validates

>>> [CLI] bundle validate
Name: databricks-ai-v5-smoke-test
Target: dev
Expand Down
2 changes: 2 additions & 0 deletions acceptance/air/get-ai-runtime/output.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@

=== get (text)

>>> [CLI] air get 123

╭─ Configuration ────────────────────────────────────────────────╮
Expand Down Expand Up @@ -35,6 +36,7 @@ Run URL: [DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]
MLflow URL: [DATABRICKS_URL]/ml/experiments/exp1/runs/run1

=== get (json)

>>> [CLI] air get 123 -o json
{
"v": 1,
Expand Down
4 changes: 4 additions & 0 deletions acceptance/air/get/output.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@

=== get (text)

>>> [CLI] air get 123

╭─ Configuration ────────────────────────────────────────────────╮
Expand Down Expand Up @@ -35,6 +36,7 @@ Run URL: [DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]
MLflow URL: [DATABRICKS_URL]/ml/experiments/exp1/runs/run1

=== get (json)

>>> [CLI] air get 123 -o json
{
"v": 1,
Expand All @@ -54,12 +56,14 @@ MLflow URL: [DATABRICKS_URL]/ml/experiments/exp1/runs/run1
}

=== invalid run id

>>> [CLI] air get notanumber
Error: invalid JOB_RUN_ID "notanumber": must be a positive integer

Exit code: 1

=== invalid run id (json)

>>> [CLI] air get notanumber -o json
{
"v": 1,
Expand Down
8 changes: 8 additions & 0 deletions acceptance/air/help/output.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@

=== help

>>> [CLI] air --help
Run and manage AI Runtime training workloads on Databricks serverless GPU compute.

Expand Down Expand Up @@ -30,6 +31,7 @@ Global Flags:
Use "databricks air [command] --help" for more information about a command.

=== list help

>>> [CLI] air list --help
List active runs for the current user.

Expand Down Expand Up @@ -66,6 +68,7 @@ Global Flags:
-t, --target string bundle target to use (if applicable)

=== get help

>>> [CLI] air get --help
Show status, configuration, and timing details for a specified run.

Expand All @@ -84,6 +87,7 @@ Global Flags:
-t, --target string bundle target to use (if applicable)

=== pools help

>>> [CLI] air pools --help
View provisioned GPU pools

Expand All @@ -107,6 +111,7 @@ Global Flags:
Use "databricks air pools [command] --help" for more information about a command.

=== images help

>>> [CLI] air images --help
Manage container images for AI Runtime

Expand All @@ -129,6 +134,7 @@ Global Flags:
Use "databricks air images [command] --help" for more information about a command.

=== images push help

>>> [CLI] air images push --help
This CLI command is in Preview. For current feature status and requirements, see:
AWS: https://docs.databricks.com/aws/en/machine-learning/ai-runtime/cli/docker-images
Expand Down Expand Up @@ -162,6 +168,7 @@ Global Flags:
-t, --target string bundle target to use (if applicable)

=== logs help

>>> [CLI] air logs --help
Stream logs from an active run until it completes, or fetch logs from a
completed run.
Expand Down Expand Up @@ -191,6 +198,7 @@ Global Flags:
-t, --target string bundle target to use (if applicable)

=== cancel help

>>> [CLI] air cancel --help
Cancel one or more runs by job run ID, or use --all to cancel
all active runs submitted by the current user.
Expand Down
10 changes: 10 additions & 0 deletions acceptance/air/image-push/output.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@

=== push image

>>> databricks air images push -p docker-test -s nvidia/cuda:13.4.1 --catalog my-catalog --schema my-schema --artifact cuda:13.4.1-cudnn-devel-ubuntu26.04
Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
Pushing nvidia/cuda:13.4.1 as my-catalog.my-schema.cuda:13.4.1-cudnn-devel-ubuntu26.04 using profile docker-test
Expand Down Expand Up @@ -35,6 +36,7 @@ tag nvidia/cuda:13.4.1 [WORKSPACE_ID].container.us-west-2.localhost/my-catalog.m
push [WORKSPACE_ID].container.us-west-2.localhost/my-catalog.my-schema.cuda:13.4.1-cudnn-devel-ubuntu26.04

=== missing image is pulled for AIR; destination defaults to source

>>> databricks air images push -p docker-test -s example/image:v1 --catalog main --schema training
Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
Pushing example/image:v1 as main.training.image:v1 using profile docker-test
Expand All @@ -50,6 +52,7 @@ Pushed image.
Now you can set `environment.unity_catalog_image = main.training.image:v1` in your YAML config and create a workload with `databricks air run -f config.yaml`.

=== forced pull and JSON output

>>> databricks air images push -p docker-test -s example/image:v1 --catalog main --schema training --pull -o json
Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
Pushing example/image:v1 as main.training.image:v1 using profile docker-test
Expand All @@ -73,6 +76,7 @@ Pushed image.
}

=== daemon failure must not pull

>>> musterr databricks air images push -p docker-test -s example/image:v1 --catalog main --schema training
Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
Pushing example/image:v1 as main.training.image:v1 using profile docker-test
Expand All @@ -83,6 +87,7 @@ Cannot connect to the Docker daemon
Error: check local source image example/image:v1 (ensure Docker is running and accessible): exit status 1

=== inspect failure must not pull

>>> musterr databricks air images push -p docker-test -s example/image:v1 --catalog main --schema training
Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
Pushing example/image:v1 as main.training.image:v1 using profile docker-test
Expand All @@ -94,6 +99,7 @@ Docker inspect failed
Error: inspect source image example/image:v1: exit status 1

=== incompatible local image must not be pushed

>>> musterr databricks air images push -p docker-test -s example/image:v1 --catalog main --schema training
Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
Pushing example/image:v1 as main.training.image:v1 using profile docker-test
Expand All @@ -104,6 +110,7 @@ Using local source image example/image:v1
Error: source image example/image:v1 has platform "linux/arm64"; AIR requires linux/amd64: use --pull to fetch a compatible image, or rebuild the source for linux/amd64

=== incompatible pulled image must not be pushed

>>> musterr databricks air images push -p docker-test -s example/image:v1 --catalog main --schema training --pull
Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
Pushing example/image:v1 as main.training.image:v1 using profile docker-test
Expand All @@ -115,6 +122,7 @@ Docker pull progress
Error: source image example/image:v1 has platform "linux/arm64"; AIR requires linux/amd64: use --pull to fetch a compatible image, or rebuild the source for linux/amd64

=== pull failure must not be pushed

>>> musterr databricks air images push -p docker-test -s example/image:v1 --catalog main --schema training --pull
Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
Pushing example/image:v1 as main.training.image:v1 using profile docker-test
Expand All @@ -126,6 +134,7 @@ Docker pull failed
Error: pull source image example/image:v1: exit status 1

=== invalid destination and removed flags

>>> musterr databricks air images push -p docker-test -s example/image:v1 --catalog main --schema training --artifact invalid.name
Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
Error: invalid artifact name "invalid.name": use lowercase letters and digits separated by one or two underscores or one or more hyphens
Expand Down Expand Up @@ -177,6 +186,7 @@ Global Flags:


=== reject oversized catalog.schema.artifact before Docker operations

>>> musterr databricks air images push -p docker-test -s example/image:v1 --catalog main --schema training --artifact aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
Error: invalid image repository name: catalog.schema.artifact must be 255 characters or less
Expand Down
Loading
Loading