Summary
Track reliability and lifecycle gaps in the aggregated BSL CA bundle used through AWS_CA_BUNDLE.
These concerns were identified while comparing the original OADP 1.5 implementation in #1969 with its OADP 1.4 backport in #2466. A review of the current oadp-dev implementation shows that the core gaps below are also still present there, although oadp-dev has added CACertRef resolution and owned Secret support.
Affected code lines
Problems to track
1. CA bundle changes do not explicitly roll out Velero
The operator updates velero-ca-bundle, but the Velero Deployment pod template references only the constant ConfigMap name. The pod template does not include a checksum/version derived from the bundle contents.
A projected ConfigMap may eventually update the mounted file, but the implementation does not establish that existing AWS SDK clients reload that file. Certificate rotation can therefore leave running Velero processes using stale trust material until another event restarts the pod.
This remains true on oadp-dev: processCACertificatesForVelero adds the volume, mount, and AWS_CA_BUNDLE, but no bundle checksum annotation is added to the pod template.
2. velero-ca-bundle is not deleted when it is no longer needed
When no applicable custom CA remains, processCACertForBSLs returns an empty name without deleting an existing managed ConfigMap. When backupImages becomes false, CA processing is skipped entirely, also leaving the previous ConfigMap behind.
The Deployment should lose its mount and environment variable when reconciled, but the stale managed resource remains. This behavior is present in #1969, #2466, and oadp-dev.
3. Invalid PEM/X.509 data is accepted after validation fails
validatePEMCertificate detects malformed or non-certificate input, but callers only log the error and still append the bytes to the generated bundle. A malformed certificate can therefore produce an unusable AWS_CA_BUNDLE while reconciliation reports success.
This behavior is also present in oadp-dev, including its generated CACert Secret path.
4. CloudStorage lookup failures can silently omit a declared CA
For a DPA CloudStorage backup location, failure to fetch the referenced CloudStorage object leaves the provider empty. The AWS-only filter then skips its CA instead of returning an actionable reconciliation error.
The same err == nil/otherwise-continue behavior remains in oadp-dev. This is distinct from CACertRef resolution, where current oadp-dev correctly returns an error for required references.
5. Scope and naming are broader than actual behavior
The feature is described as concatenating certificates from all BSLs, but it intentionally aggregates only providers whose name contains aws, because the result is exposed via AWS_CA_BUNDLE. Azure and GCP CA certificates are excluded.
This may be the correct implementation, but the contract, naming, logs, tests, and documentation should explicitly say “AWS BSL CA certificates” to avoid implying provider-neutral behavior.
Proposed behavior
- Add a deterministic hash of the final CA bundle to the Velero pod-template annotations so bundle changes trigger a rollout.
- Delete the managed
velero-ca-bundle ConfigMap when no applicable custom CA remains or CA processing is disabled.
- Define and enforce a production policy for invalid certificates:
- preferably fail reconciliation with a clear error; or
- explicitly skip invalid entries and surface a warning condition/event.
- Return a clear error when a DPA-declared CloudStorage reference cannot be resolved instead of silently omitting its CA.
- Rename/document the feature as AWS-specific unless support is deliberately expanded to other providers.
Acceptance criteria
Notes
The missing ConfigMap owner reference found specifically in #2466 is not included here because it is a backport-only parity problem: #1969 and oadp-dev set the DPA controller reference. It should be fixed directly in #2466 rather than tracked as a shared issue.
Related historical work: #1834, #2097, and #2384.
Summary
Track reliability and lifecycle gaps in the aggregated BSL CA bundle used through
AWS_CA_BUNDLE.These concerns were identified while comparing the original OADP 1.5 implementation in #1969 with its OADP 1.4 backport in #2466. A review of the current
oadp-devimplementation shows that the core gaps below are also still present there, althoughoadp-devhas addedCACertRefresolution and owned Secret support.Affected code lines
oadp-1.4: [oadp-1.4] OADP-8835: feat(bsl): concatenate all CA certificates from BSLs and include system defaults #2466oadp-1.5: oadp-1.5: OADP-6765: feat(bsl): concatenate all CA certificates from BSLs and include system defaults #1969oadp-dev:internal/controller/bsl.go(processCACertForBSLs) andinternal/controller/velero.go(processCACertificatesForVelero)Problems to track
1. CA bundle changes do not explicitly roll out Velero
The operator updates
velero-ca-bundle, but the Velero Deployment pod template references only the constant ConfigMap name. The pod template does not include a checksum/version derived from the bundle contents.A projected ConfigMap may eventually update the mounted file, but the implementation does not establish that existing AWS SDK clients reload that file. Certificate rotation can therefore leave running Velero processes using stale trust material until another event restarts the pod.
This remains true on
oadp-dev:processCACertificatesForVeleroadds the volume, mount, andAWS_CA_BUNDLE, but no bundle checksum annotation is added to the pod template.2.
velero-ca-bundleis not deleted when it is no longer neededWhen no applicable custom CA remains,
processCACertForBSLsreturns an empty name without deleting an existing managed ConfigMap. WhenbackupImagesbecomesfalse, CA processing is skipped entirely, also leaving the previous ConfigMap behind.The Deployment should lose its mount and environment variable when reconciled, but the stale managed resource remains. This behavior is present in #1969, #2466, and
oadp-dev.3. Invalid PEM/X.509 data is accepted after validation fails
validatePEMCertificatedetects malformed or non-certificate input, but callers only log the error and still append the bytes to the generated bundle. A malformed certificate can therefore produce an unusableAWS_CA_BUNDLEwhile reconciliation reports success.This behavior is also present in
oadp-dev, including its generated CACert Secret path.4. CloudStorage lookup failures can silently omit a declared CA
For a DPA
CloudStoragebackup location, failure to fetch the referencedCloudStorageobject leaves the provider empty. The AWS-only filter then skips its CA instead of returning an actionable reconciliation error.The same
err == nil/otherwise-continue behavior remains inoadp-dev. This is distinct fromCACertRefresolution, where currentoadp-devcorrectly returns an error for required references.5. Scope and naming are broader than actual behavior
The feature is described as concatenating certificates from all BSLs, but it intentionally aggregates only providers whose name contains
aws, because the result is exposed viaAWS_CA_BUNDLE. Azure and GCP CA certificates are excluded.This may be the correct implementation, but the contract, naming, logs, tests, and documentation should explicitly say “AWS BSL CA certificates” to avoid implying provider-neutral behavior.
Proposed behavior
velero-ca-bundleConfigMap when no applicable custom CA remains or CA processing is disabled.Acceptance criteria
backupImages: falsecleans up CA-bundle resources and Deployment wiring.AWS_CA_BUNDLE.oadp-devand supported release branches containing oadp-1.5: OADP-6765: feat(bsl): concatenate all CA certificates from BSLs and include system defaults #1969/[oadp-1.4] OADP-8835: feat(bsl): concatenate all CA certificates from BSLs and include system defaults #2466.Notes
The missing ConfigMap owner reference found specifically in #2466 is not included here because it is a backport-only parity problem: #1969 and
oadp-devset the DPA controller reference. It should be fixed directly in #2466 rather than tracked as a shared issue.Related historical work: #1834, #2097, and #2384.