Is your feature request related to a problem? Please describe.
In centralized agent mode (agent.managementCluster: true), the Helm chart does not render the per-cluster sveltos-agent and drift-detection-manager Deployments. Classifier and addon-controller generate these Deployments at run time from a static manifest.
A SveltosCluster with spec.workloadIdentity.provider: Azure needs the pod label azure.workload.identity/use: "true" on its agent. The azure-workload-identity webhook uses an objectSelector for this label. Without the label, the webhook does not mutate the pod, and the agent does not get AZURE_FEDERATED_TOKEN_FILE. The agent then panics at start:
failed to get secret" err="AZURE_FEDERATED_TOKEN_FILE env var is not set; ensure the pod is configured with Azure Workload Identity"
panic: 1
The patch ConfigMap (agentPatchConfigMap and driftDetectionManagerPatchConfigMap) is the only method to add the label today. This method works, but a user cannot limit the patch to the Azure clusters.
A patch entry without a target gets the default selector Kind: Deployment, Group: apps. The patch then applies to every generated agent Deployment. In our management cluster, the patch also applied to 3 CAPI agents and to 1 agent for a SveltosCluster that uses a kubeconfig. The label is inert for those clusters, but the webhook mounts an Azure identity token into 4 pods that never read the token.
A target cannot solve this problem, because the generated Deployment does not show the provider. Classifier and addon-controller add only these labels: cluster-name, cluster-namespace, cluster-type, and feature. The only available workaround is a labelSelector on cluster-namespace. This workaround depends on a local convention, because it assumes that all clusters in one namespace use the same provider.
Describe the solution you'd like
Add one label to the generated Deployment that shows the workload identity provider. For example:
workload-identity-provider: azure
Set the label from spec.workloadIdentity.provider, with a lowercase value: azure, aws, gcp, or oidc. Omit the label when the SveltosCluster uses a kubeconfig, because then no provider applies. Add the label in classifier (sveltos-agent), in addon-controller (drift-detection-manager), and in sveltos-applier.
A patch can then select the correct clusters, and the documentation can show a complete example:
classifierManager:
agentPatchConfigMap:
data:
azure-workload-identity: |
patch: |
- op: add
path: /spec/template/metadata/labels/azure.workload.identity~1use
value: "true"
target:
group: apps
kind: Deployment
labelSelector: workload-identity-provider=azureThe label is not part of the CRD, therefore this solution does not add API surface.
One constraint applies to the implementation. Do not add the new label to the map that getSveltosAgentLabels returns. getSveltosAgentDeploymentName uses that map as an exact MatchLabels selector to find the existing Deployment. For zero matches, getInstantiatedObjectName creates a new random name and registers the name with the key manager.
An extra key in the map makes every existing agent Deployment stop to match after an upgrade. The controller then creates a second Deployment and leaves the first Deployment without an owner. The reconcile can also fail, because the key manager permits one name for each managed cluster. The same constraint applies to getDriftDetectionManagerLabels in addon-controller. Apply the provider label to the rendered object only, and keep the discovery labels unchanged.
deploySveltosAgentInManagementCluster already receives c client.Client, and getSveltosAgentPatches already calls clusterproxy.GetCluster for the per-cluster patch annotation. A read of spec.workloadIdentity.provider at this point is cheap.
Describe alternatives you've considered
- Add
azure.workload.identity/use: "true" automatically when the provider is Azure. This alternative needs no user configuration, but it hides the behavior, and it keeps the dependency on the azure-workload-identity webhook.
- Add a field to the
SveltosCluster CRD, for example an Azure opt-in flag or a general spec.deploymentTemplate. This alternative grows the CRD for each new pod setting.
- Inject the credentials directly, and do not use the webhook. Classifier and addon-controller can add the projected
serviceAccountToken volume with the audience api://AzureADTokenExchange and the environment variable AZURE_FEDERATED_TOKEN_FILE. getAzureRestConfig already reads clientID and tenantID from the spec, therefore the token path is the only external input. This alternative also permits a different identity for each cluster on AWS, because IRSA reads roleARN from the ServiceAccount today. The change is larger than a label.
- Select on
cluster-namespace in the patch target. This alternative works today, but it encodes a local convention.
- Add the label with an external mutation policy, for example Kyverno. This alternative adds one more component to the management cluster.
Option 3 and this request are not exclusive. A later change can add the credentials automatically and keep the label for other patches.
Additional context
Environment: chart 1.15.0, centralized agent mode, 12 agents in one management cluster. 8 agents belong to SveltosCluster instances with spec.workloadIdentity.provider: Azure. Those 8 agents restarted continuously from the first reconcile until we added the patch ConfigMap.
Two more results from this work can help the documentation:
Related: projectsveltos/helm-charts#220.
Is your feature request related to a problem? Please describe.
In centralized agent mode (
agent.managementCluster: true), the Helm chart does not render the per-clustersveltos-agentanddrift-detection-managerDeployments. Classifier and addon-controller generate these Deployments at run time from a static manifest.A
SveltosClusterwithspec.workloadIdentity.provider: Azureneeds the pod labelazure.workload.identity/use: "true"on its agent. The azure-workload-identity webhook uses anobjectSelectorfor this label. Without the label, the webhook does not mutate the pod, and the agent does not getAZURE_FEDERATED_TOKEN_FILE. The agent then panics at start:The patch ConfigMap (
agentPatchConfigMapanddriftDetectionManagerPatchConfigMap) is the only method to add the label today. This method works, but a user cannot limit the patch to the Azure clusters.A patch entry without a
targetgets the default selectorKind: Deployment, Group: apps. The patch then applies to every generated agent Deployment. In our management cluster, the patch also applied to 3 CAPI agents and to 1 agent for aSveltosClusterthat uses a kubeconfig. The label is inert for those clusters, but the webhook mounts an Azure identity token into 4 pods that never read the token.A
targetcannot solve this problem, because the generated Deployment does not show the provider. Classifier and addon-controller add only these labels:cluster-name,cluster-namespace,cluster-type, andfeature. The only available workaround is alabelSelectoroncluster-namespace. This workaround depends on a local convention, because it assumes that all clusters in one namespace use the same provider.Describe the solution you'd like
Add one label to the generated Deployment that shows the workload identity provider. For example:
Set the label from
spec.workloadIdentity.provider, with a lowercase value:azure,aws,gcp, oroidc. Omit the label when theSveltosClusteruses a kubeconfig, because then no provider applies. Add the label in classifier (sveltos-agent), in addon-controller (drift-detection-manager), and insveltos-applier.A patch can then select the correct clusters, and the documentation can show a complete example:
The label is not part of the CRD, therefore this solution does not add API surface.
One constraint applies to the implementation. Do not add the new label to the map that
getSveltosAgentLabelsreturns.getSveltosAgentDeploymentNameuses that map as an exactMatchLabelsselector to find the existing Deployment. For zero matches,getInstantiatedObjectNamecreates a new random name and registers the name with the key manager.An extra key in the map makes every existing agent Deployment stop to match after an upgrade. The controller then creates a second Deployment and leaves the first Deployment without an owner. The reconcile can also fail, because the key manager permits one name for each managed cluster. The same constraint applies to
getDriftDetectionManagerLabelsin addon-controller. Apply the provider label to the rendered object only, and keep the discovery labels unchanged.deploySveltosAgentInManagementClusteralready receivesc client.Client, andgetSveltosAgentPatchesalready callsclusterproxy.GetClusterfor the per-cluster patch annotation. A read ofspec.workloadIdentity.providerat this point is cheap.Describe alternatives you've considered
azure.workload.identity/use: "true"automatically when the provider is Azure. This alternative needs no user configuration, but it hides the behavior, and it keeps the dependency on the azure-workload-identity webhook.SveltosClusterCRD, for example an Azure opt-in flag or a generalspec.deploymentTemplate. This alternative grows the CRD for each new pod setting.serviceAccountTokenvolume with the audienceapi://AzureADTokenExchangeand the environment variableAZURE_FEDERATED_TOKEN_FILE.getAzureRestConfigalready readsclientIDandtenantIDfrom the spec, therefore the token path is the only external input. This alternative also permits a different identity for each cluster on AWS, because IRSA readsroleARNfrom the ServiceAccount today. The change is larger than a label.cluster-namespacein the patch target. This alternative works today, but it encodes a local convention.Option 3 and this request are not exclusive. A later change can add the credentials automatically and keep the label for other patches.
Additional context
Environment: chart 1.15.0, centralized agent mode, 12 agents in one management cluster. 8 agents belong to
SveltosClusterinstances withspec.workloadIdentity.provider: Azure. Those 8 agents restarted continuously from the first reconcile until we added the patch ConfigMap.Two more results from this work can help the documentation:
creationTimestampvalue did not change, and thegenerationvalue increased. Comment 1 in Centralized Agent Mode: dynamically-created per-cluster sveltos-agent Deployment still missing azure.workload.identity/use label (not covered by #218/#219) helm-charts#220 reports a different result for chart 1.13.1, where the patch applied only at creation time. The config hash includes the patch content now.getSveltosAgentPatchesreturns early for a non-nil per-cluster result, therefore a user who sets both ConfigMaps loses the global patches.Related: projectsveltos/helm-charts#220.