diff --git a/docs.json b/docs.json index 74606b2a3..6f4058f57 100644 --- a/docs.json +++ b/docs.json @@ -423,6 +423,7 @@ "runpodctl/reference/runpodctl-pod", "runpodctl/reference/runpodctl-serverless", "runpodctl/reference/runpodctl-template", + "runpodctl/reference/runpodctl-model", "runpodctl/reference/runpodctl-hub", "runpodctl/reference/runpodctl-network-volume", "runpodctl/reference/runpodctl-registry", diff --git a/runpodctl/overview.mdx b/runpodctl/overview.mdx index 8fa4d6512..48fa20934 100644 --- a/runpodctl/overview.mdx +++ b/runpodctl/overview.mdx @@ -181,6 +181,7 @@ Runpod CLI organizes commands into groups based on the resource type: | `runpodctl pod` | | Manage Pods (create, list, start, stop, delete) | | `runpodctl serverless` | `sls` | Manage Serverless endpoints | | `runpodctl template` | `tpl` | List, search, and manage templates | +| `runpodctl model` | | Manage models in the model repository (add, list, remove) | | `runpodctl hub` | | Browse and deploy from the Runpod Hub | | `runpodctl network-volume` | `nv` | Manage network volumes | | `runpodctl registry` | `reg` | Manage container registry authentications | diff --git a/runpodctl/reference/runpodctl-config.mdx b/runpodctl/reference/runpodctl-config.mdx index c4644628b..70b636b54 100644 --- a/runpodctl/reference/runpodctl-config.mdx +++ b/runpodctl/reference/runpodctl-config.mdx @@ -35,6 +35,22 @@ Your Runpod API key, which authenticates the CLI to access your account. You can The Runpod API endpoint URL. The default value should work for most users. +## Environment variables + +You can also configure the CLI using environment variables. When set, these take precedence over the corresponding values in your config file. + + +Your Runpod API key. Overrides the `apiKey` value in your config file. + + + +The Runpod GraphQL API endpoint URL. Overrides the `apiUrl` value in your config file. + + + +The Runpod REST API endpoint URL. Overrides the `restApiUrl` value in your config file. + + ## Related commands - [`runpodctl doctor`](/runpodctl/reference/runpodctl-doctor) diff --git a/runpodctl/reference/runpodctl-model.mdx b/runpodctl/reference/runpodctl-model.mdx new file mode 100644 index 000000000..4f41975c5 --- /dev/null +++ b/runpodctl/reference/runpodctl-model.mdx @@ -0,0 +1,159 @@ +--- +title: "model" +sidebarTitle: "model" +--- + +Manage models in the Runpod model repository. Use these subcommands to upload model files, list your models, and remove models or individual model versions. + + +```bash Command +runpodctl model [flags] +``` + + +## Subcommands + +### Add a model + +Add a model to the model repository. You can add a model in two ways: by uploading model files from your local machine, or by mirroring an existing Hugging Face model server-side. + +To upload model files from your local machine, point `--model-path` at the directory that contains them: + +```bash +runpodctl model add --name "my-model" --owner "my-team" --model-path ./model +``` + +When you upload from a directory in a terminal, `runpodctl` shows a progress bar with upload progress and estimated time remaining, then prints the total model size once the upload finishes. + +If you reuse one local directory across model versions without this flag, a later upload can include a mix of old and new files. Add `--delete-my-model-files-after-upload` to remove exactly the files this run uploaded from `--model-path` once the model version is confirmed deployable server-side. It never touches unrelated files, and it runs only after the version is confirmed deployable, so an upload that fails or whose deployable confirmation times out leaves your files in place. This flag requires both `--model-path` and `--wait-for-hash`. + +```bash +runpodctl model add --name "my-model" --owner "my-team" --model-path ./model --wait-for-hash --delete-my-model-files-after-upload +``` + + + +If your account has a model repository storage quota, `runpodctl model add` checks the total upload size against your remaining quota before it creates the model or any upload session. An upload that would exceed the quota fails immediately, and the error reports how much space is available. To proceed, free up space or contact support to request a higher quota. + +This check applies only to local uploads. Mirroring a Hugging Face model with `--huggingface-model` has no local upload session, so the check does not apply. + + + +To mirror an existing Hugging Face model into the Runpod model repository, use `--huggingface-model` with the source model in `owner/repo` form. `--name` sets the destination model name in the Runpod repository, and `--owner` is optional when mirroring. + +The transfer runs server-side, so there is no local upload session, file upload, or progress bar. Runpod creates the new model version right away and transfers the files from Hugging Face in the background. + +Mirroring cannot be combined with the local-upload flags (`--model-path`, `--create-upload`, `--file-name`, `--file-size`, `--part-size`, `--content-type`, and `--wait-for-hash`). Uploading and mirroring are two separate ways to add a model. + +```bash +runpodctl model add --name tiny-llm --huggingface-model arnir0/Tiny-LLM +``` + +#### Add flags + + +Model name. + + + +Model owner namespace (a user or team owner ID). + + + +Directory containing the model files to upload. + + + +Hugging Face model to mirror into the Runpod model repository, in `owner/repo` form. Cannot be combined with the local-upload flags (`--model-path`, `--create-upload`, `--file-name`, `--file-size`, `--part-size`, `--content-type`, and `--wait-for-hash`). + + + +Wait until the uploaded model version is deployable before the command returns; despite the flag name, the wait is for deployability, not just for the file hash. Requires `--model-path`. Once the version is confirmed deployable, `runpodctl` prints a confirmation message, along with the model's URL. + + + +Maximum time to wait when `--wait-for-hash` is set. Set to `0` to wait indefinitely. + + + +Delete the uploaded `--model-path` files after the model version is confirmed deployable server-side. Only the files this run uploaded are deleted, including nested subdirectories, and only after the version is confirmed deployable. If that confirmation times out or fails, no files are deleted. + +Requires both `--model-path` and `--wait-for-hash`. Both are validated before the upload begins, so the command fails immediately if either is missing. + +If a specific file cannot be deleted, the command exits non-zero and names that file, so there is no silent partial success. A stderr confirmation line prints once at least one file has been deleted. The `deletedModelFiles` (count) and `deletedModelFilesBytes` (bytes) fields are added to the JSON output only when `--verbose`, `-v` is set. + + + +Include the full upload details in the `--wait-for-hash` output. Without this flag, a compact summary is printed instead. + + + + +`runpodctl model add` also accepts lower-level flags for configuring uploads directly, such as `--content-type`, `--file-name`, `--file-size`, `--part-size`, `--metadata`, and `--model-status`. Run `runpodctl model add --help` to see the complete list. + + + +### List models + +List the models in your model repository: + +```bash +runpodctl model list +``` + +#### List flags + + +Filter the results by model name. + + + +Filter the results by provider. + + +### Remove a model + +Remove a model from the model repository: + +```bash +runpodctl model remove --name "my-model" --owner "my-team" +``` + +To remove a single model version instead of the entire model, pass either `--hash` or `--version`. This marks that version as removed and leaves the rest of the model in place. + +```bash +runpodctl model remove --name "my-model" --owner "my-team" --version "" +``` + + + +`runpodctl model remove` refuses to remove a model or version that a Serverless endpoint still references. Removing the whole model is refused if any version is referenced; removing a specific version with `--hash` or `--version` is refused only when that exact version is referenced. The command reports which endpoints block the removal, and nothing is deleted when a removal is refused. + +There is no override or force flag. A model or version in active use cannot be removed until you detach or replace it on the referencing endpoint. If the dependency check cannot complete, for example when the API is unreachable, the removal is also refused, because Runpod treats an unverifiable dependency check as unsafe. + + + +To clear a blocked removal, detach or replace the model on the referencing endpoint, then retry the removal. Detach the model with `runpodctl serverless update --clear-models`, or replace it with `runpodctl serverless update --model-reference `. See the [Update an endpoint](/runpodctl/reference/runpodctl-serverless#update-an-endpoint) section for details. + +#### Remove flags + + +Model name. + + + +Model owner. + + + +Hash of the model version to remove. Cannot be combined with `--version`. + + + +UUID of the model version to remove. Cannot be combined with `--hash`. + + +## Related commands + +- [`runpodctl serverless`](/runpodctl/reference/runpodctl-serverless) +- [`runpodctl config`](/runpodctl/reference/runpodctl-config) diff --git a/runpodctl/reference/runpodctl-serverless.mdx b/runpodctl/reference/runpodctl-serverless.mdx index 7daaeaff6..7ef8bd4d7 100644 --- a/runpodctl/reference/runpodctl-serverless.mdx +++ b/runpodctl/reference/runpodctl-serverless.mdx @@ -165,6 +165,12 @@ Update endpoint configuration: ```bash runpodctl serverless update --workers-max 5 + +# Cache a model reference on the endpoint +runpodctl serverless update --model-reference https://huggingface.co//: + +# Clear all model references from the endpoint +runpodctl serverless update --clear-models ``` #### Update flags @@ -205,6 +211,18 @@ Enable or disable flash boot for faster worker startup. Execution timeout in seconds. Jobs that exceed this duration are terminated. +Attaching model references caches the referenced models on the endpoint so workers start faster and cost less. See [Model caching](/serverless/endpoints/model-caching) for when and why to cache models. + + +Model reference URL to cache on the endpoint, in the same form used by `serverless create` (a Hugging Face reference like `https://huggingface.co//:`). Repeatable: use multiple `--model-reference` flags to cache several models. Replaces the endpoint's existing model references. Cannot be combined with `--clear-models`. + + + +Remove all model references from the endpoint. Cannot be combined with `--model-reference`. + + +Updating an endpoint's model references with `--model-reference` or `--clear-models` works on both GPU and CPU endpoints. On a GPU endpoint, the update preserves the endpoint's GPU selection, including any excluded GPU types, rather than widening it. The command output includes the endpoint's current model references in the `modelReferences` field, which is an empty array after `--clear-models`. + ### Delete an endpoint Delete an endpoint: