Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 35 additions & 5 deletions internal/system/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ d8 system (aliases: s, p, platform)
│ └── main Dump the main queue
├── logs Stream deckhouse-controller logs
└── collect-debug-info Stream a gzipped debug tarball to stdout
└── virtualization Stream a d8-virtualization-only debug tarball
```

The `s` alias is the recommended short form (`d8 s module list`). `p` and `platform` are legacy aliases kept for backward compatibility with older documentation.
Expand Down Expand Up @@ -227,16 +228,40 @@ Collects a wide cluster snapshot into a **gzipped tar streamed to stdout**, so y
d8 system collect-debug-info > deckhouse-debug-$(date +"%Y_%m_%d").tar.gz
```

It refuses to run when stdout is a terminal (to avoid dumping binary to your screen) unless you pass `--list-exclude`. The collection runs **inside** the leader pod: it executes on the order of ~60 diagnostic commands there (`deckhouse-controller queue list`, redacted global values, module/source/release inventories, cluster-wide `kubectl get` snapshots, and controller/etcd/apiserver/VPA/Prometheus logs, plus cloud-provider/cert-manager/istio/cni-cilium extras when those modules are Ready), writing each result as a file in the archive.
It refuses to run when stdout is a terminal (to avoid dumping binary to your screen) unless you pass `--list-exclude`. The collection runs **inside** the leader pod: it executes the 63 declared commands there (`deckhouse-controller queue list`, redacted global values, module/source/release inventories, cluster-wide `kubectl get` snapshots, and controller/etcd/apiserver/VPA/Prometheus logs, plus cloud-provider/cert-manager/istio/cni-cilium/virtualization extras when those modules are Ready), writing each result as a file in the archive. The two per-cloud log collections are repeated once per matching provider module, so a cloud cluster ends up with slightly more archive entries than commands.

Before collecting, the command reads the list of `Ready` modules to decide which module-gated commands apply. If that read fails, it prints an error and keeps going: module-gated commands run anyway (and may produce empty files), except the per-module log collections whose file name contains the module name - those are skipped, since their archive entry name cannot be resolved.

| Flag | Short | Type | Default | Description |
|---|---|---|---|---|
| `--exclude` | | string list | (none) | Comma-separated list of elements to leave out of the archive. Matches by base name, so e.g. `ccm-logs` also drops the per-cloud `ccm-logs-<module>.txt`. |
| `--list-exclude` | `-l` | bool | `false` | Print the names of everything that can be excluded, then exit. This path makes no cluster calls. |
| `--exclude` | | string list | (none) | Comma-separated list of entries to leave out of the archive. Accepts exactly the names printed by `--list-exclude`, with or without the file extension and ignoring surrounding spaces; a name matches that entry only, never a group of files sharing a prefix. A name that matches nothing is an error listing close matches, so a typo cannot pass as "collect everything". |
| `--list-exclude` | `-l` | bool | `false` | Print the names accepted by `--exclude`, then exit. This path makes no cluster calls. The names are the archive file names as declared in the command table, except the per-module cloud logs, which are printed as the module-independent keys `ccm-logs` and `csi-controller-logs` (their real entry is `d8-<module>-ccm-logs.txt`, and both spellings are accepted). |
| `--command-timeout` | | duration | `2m` | Timeout applied to each individual in-pod command. |
| `--request-interval` | | duration | `0` | Minimum gap between commands to avoid overloading the cluster (e.g. `200ms`, `1s`). `0` disables rate limiting. |

**Handle the archive as sensitive.** Only `global-values.json` is redacted (its `kubeRBACProxyCA` and registry `dockercfg`); container logs and the raw `audit-policy` Secret are included unredacted. Also note that a file is written even when its source command fails or times out, so an entry may be empty rather than absent.
While collecting, the command prints only its start and completion banners to stderr; individual entries are not announced. Failures are the exception - they are reported as they happen.

A file is written even when its source command fails or times out, so an entry may be empty or truncated rather than absent. Every such command is listed in a **`collection-errors.txt`** entry added to the archive, naming the entry, the command, the error (or the timeout) and how many bytes were kept. The archive has no `collection-errors.txt` when everything succeeded. Check for it before treating an empty entry as "the resource holds nothing" - the warnings printed during collection go to stderr, which is not part of the archive.

**Handle the archive as sensitive.** Only `cluster-global-values.json` is redacted (its `kubeRBACProxyCA` and registry `dockercfg`); container logs and the raw audit policy Secret (`kube-system-audit-policy.json`) are included unredacted.

### `collect-debug-info virtualization`

Collects a separate, virtualization-focused archive: the pod list of the `d8-virtualization` namespace plus the **full** log of every pod in it (`--tail=-1`, no line cap). Same stdout rules as the parent command.

The pod list is read through the Kubernetes API with **your own** kubeconfig, so the account you run `d8` with needs `list pods` in `d8-virtualization`; the logs themselves are still collected by `kubectl` running inside the leader pod, like every other archive entry.

```bash
d8 system collect-debug-info virtualization > deckhouse-debug-virtualization-$(date +"%Y_%m_%d").tar.gz
```

| Flag | Short | Type | Default | Description |
|---|---|---|---|---|
| `--skip-ds-logs` | | bool | `false` | Skip logs of pods owned by a DaemonSet (`virt-handler`, `virtualization-dra`, `vm-route-forge`, ...), whose volume scales with the number of nodes. |
| `--command-timeout` | | duration | `2m` | Timeout applied to each individual in-pod command, and to the pod list request. |
| `--request-interval` | | duration | `0` | Minimum gap between commands to avoid overloading the cluster. |

The pod list is the entire payload of this archive, so the command fails (and writes nothing) when the namespace cannot be listed or holds no pods - instead of producing a valid-looking archive with a single empty file. `--exclude`/`--list-exclude` do not apply here.

---

Expand Down Expand Up @@ -312,6 +337,10 @@ d8 system collect-debug-info --list-exclude
d8 system collect-debug-info --exclude ccm-logs,csi-controller-logs \
> deckhouse-debug-$(date +"%Y_%m_%d").tar.gz

# Collect the virtualization-only archive, without DaemonSet pod logs
d8 system collect-debug-info virtualization --skip-ds-logs \
> deckhouse-debug-virtualization-$(date +"%Y_%m_%d").tar.gz


# --- Global flags ---

Expand All @@ -328,5 +357,6 @@ d8 system --kubeconfig ~/.kube/prod.config --context prod module list
- **`approve` / `apply-now` are annotation-only and idempotent.** They never error on an already-annotated or non-`Pending` release; they print a notice and exit 0.
- **`package scan` does not scan locally and does not wait.** It creates a `PackageRepositoryOperation` and returns; results are reported by the platform, not the CLI.
- **In-pod commands need a leader pod.** `module list`/`values`/`snapshots`, `queue`, and `collect-debug-info` exec into the pod labeled `leader=true` in `d8-system`; without it they fail with `no pods deckhouse available in namespace d8-system`.
- **The debug archive is sensitive** (unredacted logs and the raw audit-policy Secret) and must be redirected to a file.
- **The debug archive is sensitive** (unredacted logs and the raw audit policy Secret, `kube-system-audit-policy.json`) and must be redirected to a file.
- **`collect-debug-info` takes no positional arguments.** A misspelled subcommand (`virtualisation`) is rejected with `unknown command` instead of silently running the full cluster-wide collection.
- **stdout vs stderr:** `module` state changes print to stdout while notices/warnings/errors print to stderr, which makes it easy to script against applied changes only.
22 changes: 8 additions & 14 deletions internal/system/cmd/collect-debug-info/collect-debug-info.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ import (
"k8s.io/kubectl/pkg/util/templates"

"github.com/deckhouse/deckhouse-cli/internal/system/cmd/collect-debug-info/debugtar"
"github.com/deckhouse/deckhouse-cli/internal/system/cmd/collect-debug-info/virtualizationtar"
"github.com/deckhouse/deckhouse-cli/internal/utilk8s"
)

Expand Down Expand Up @@ -57,6 +58,7 @@ func NewCommand() *cobra.Command {
Short: "Collect debug info.",
Long: collectDebugInfoCmdLong,
Example: collectDebugInfoCmdExample,
Args: cobra.NoArgs,
SilenceErrors: true,
SilenceUsage: true,
PreRunE: func(_ *cobra.Command, _ []string) error {
Expand All @@ -74,11 +76,13 @@ func NewCommand() *cobra.Command {
return collectDebugInfo(cmd, listExclude, excludeList, commandTimeout, requestInterval)
},
}
collectDebugInfoCmd.Flags().StringSliceVar(&excludeList, "exclude", []string{}, "Exclude specific files from the debug archive. Use comma-separated values")
collectDebugInfoCmd.Flags().BoolVarP(&listExclude, "list-exclude", "l", false, "List all files that can be excluded from the debug archive")
collectDebugInfoCmd.Flags().StringSliceVar(&excludeList, "exclude", []string{}, "Exclude specific files from the debug archive. Use comma-separated names as printed by --list-exclude; the file extension is optional. An unknown name is an error")
collectDebugInfoCmd.Flags().BoolVarP(&listExclude, "list-exclude", "l", false, "List all files that can be excluded from the debug archive, then exit")
collectDebugInfoCmd.Flags().DurationVar(&commandTimeout, "command-timeout", 2*time.Minute, "Timeout for each individual debug command execution")
collectDebugInfoCmd.Flags().DurationVar(&requestInterval, "request-interval", 0, "Minimum interval between debug command executions to avoid overloading the cluster (e.g. 200ms, 500ms, 1s). Zero disables rate limiting (default 0s)")

collectDebugInfoCmd.AddCommand(virtualizationtar.NewCommand())
Comment thread
VaLosev marked this conversation as resolved.

Comment thread
VaLosev marked this conversation as resolved.
return collectDebugInfoCmd
}

Expand All @@ -96,19 +100,9 @@ func collectDebugInfo(cmd *cobra.Command, listExclude bool, excludeList []string
return nil
}

kubeconfigPath, err := cmd.Flags().GetString("kubeconfig")
if err != nil {
return fmt.Errorf("Failed to setup Kubernetes client: %w", err)
}

contextName, err := cmd.Flags().GetString("context")
if err != nil {
return fmt.Errorf("Failed to setup Kubernetes client: %w", err)
}

config, kubeCl, err := utilk8s.SetupK8sClientSet(kubeconfigPath, contextName)
config, kubeCl, err := utilk8s.NewClientSet(cmd)
if err != nil {
return fmt.Errorf("Failed to setup Kubernetes client: %w", err)
return err
}

if err = debugtar.Tarball(config, kubeCl, excludeList, commandTimeout, requestInterval); err != nil {
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
package collectdebuginfo

import (
"io"
"strings"
"testing"

"github.com/spf13/cobra"
)

// TestSubcommandTypoIsRejected guards the Args: cobra.NoArgs on the parent
// command. Without it cobra accepts an unknown positional argument silently
// (its "unknown command" check only fires for the root command) and the full
// cluster-wide collection runs instead of the requested subcommand.
func TestSubcommandTypoIsRejected(t *testing.T) {
root := &cobra.Command{Use: "d8", SilenceErrors: true, SilenceUsage: true}
system := &cobra.Command{Use: "system"}
root.AddCommand(system)
system.AddCommand(NewCommand())

root.SetOut(io.Discard)
root.SetErr(io.Discard)
root.SetArgs([]string{"system", "collect-debug-info", "virtualisation"})

err := root.Execute()
if err == nil {
t.Fatal("a misspelled subcommand was accepted, the full collection would have run")
}

if !strings.Contains(err.Error(), "unknown command") {
t.Errorf("error = %v, want it to mention an unknown command", err)
}
}
Loading
Loading