Skip to content

Fix proxysql-ca.pem leaking key of intermediate CA - #2797

Open
seunghun1ee wants to merge 1 commit into
stackhpc/2026.1from
fix-proxysql-ca-file
Open

seunghun1ee wants to merge 1 commit into
stackhpc/2026.1from
fix-proxysql-ca-file

Conversation

@seunghun1ee

Copy link
Copy Markdown
Member

The playbook secret-store-generate-internal-tls.yml has been creating proxysql-ca.pem by copying OS-TLS-INT.pem. This file contains key of intermediate CA, resulting leaking the key to controller hosts if ProxySQL is enabled.

Fixed by changing the source to OS-TLS-INT.crt which only has the certificate of intermediate CA.

Added playbook and documentation for rotating intermediate CA issuer.

@seunghun1ee seunghun1ee self-assigned this Nov 16, 2026
@seunghun1ee
seunghun1ee requested a review from a team as a code owner November 16, 2026 16:32
@github-actions github-actions Bot added the waiting-review PR is waiting for a review label Nov 16, 2026
@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Explain Change →

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
Walkthrough

The ProxySQL CA copy task now reads OS-TLS-INT.crt. A new playbook disables the OS-TLS-INT engine on the seed and controllers, then imports secret-store deployment playbooks. The configuration guide and release note describe rotation and certificate regeneration.

Changes: Intermediate CA handling

Layer / File(s) Summary
ProxySQL CA source and release note
etc/kayobe/ansible/secret-store/secret-store-generate-internal-tls.yml, releasenotes/notes/fix-proxysql-ca-leaking-int-ca-key-1582ce565a584ef3.yaml
The ProxySQL CA copy task uses OS-TLS-INT.crt as its source. The release note describes the reported intermediate CA key leak and the rotation and certificate-regeneration steps.
Issuer rotation playbook and instructions
etc/kayobe/ansible/secret-store/secret-store-rotate-intermediate-issuer.yml, doc/source/configuration/openbao.rst
The playbook disables the OS-TLS-INT engine on the seed and controllers, then imports the seed and overcloud deployment playbooks. The guide describes issuer rotation, re-encrypting OS-TLS-INT.pem, and regenerating and deploying TLS certificates.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Rotation as Rotation playbook
  participant Seed as Seed OpenBao
  participant Controllers as Controller OpenBao
  participant Deployment as Secret-store deployment playbooks
  Rotation->>Seed: Disable OS-TLS-INT using the root token
  Rotation->>Controllers: Disable OS-TLS-INT using the root token
  Rotation->>Deployment: Import seed and overcloud deployment playbooks
Loading
Merge Risk: Moderate · previous review a8e0a

The change stops new deployments from copying the intermediate CA key to controllers. The suggested remediation for existing systems does not clearly remove trust in the exposed key, and the rotation instructions can leave the regenerated key unencrypted for some layouts. Tighten the release note and documentation before merging.

Pre-merge checks: Passed 2

Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 112ac66 and a8e0a5e.

Files selected for processing: 4

  • doc/source/configuration/openbao.rst
  • etc/kayobe/ansible/secret-store/secret-store-generate-internal-tls.yml
  • etc/kayobe/ansible/secret-store/secret-store-rotate-intermediate-issuer.yml
  • releasenotes/notes/fix-proxysql-ca-leaking-int-ca-key-1582ce565a584ef3.yaml

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

Comment on lines +224 to +245
Rotating intermediate CA issuer
-------------------------------

Users can rotate the intermediate CA issuer with
``secret-store-rotate-intermediate-issuer.yml`` playbook.

.. code-block:: bash

kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/secret-store/secret-store-rotate-intermediate-issuer.yml

This will regenerate intermediate CA certificate files ``OS-TLS-INT.crt`` and
``OS-TLS-INT.pem``.
Make sure to encrypt ``OS-TLS-INT.pem`` again.

.. code-block:: bash

ansible-vault encrypt --vault-password-file ~/vault.pass $KAYOBE_CONFIG_PATH/openbao/OS-TLS-INT.pem

Once new issuer is set, regenerate TLS certificates by following instructions
at :ref:`Certificates generation<certificates-generation>`, then deploy them
by following instructions at
:ref:`Certificates deployment<openbao-certificates>`.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor · Security & Privacy · Quick win

Add the vault and environment paths to the rotation encrypt step, and warn that rotation disables the old issuer.

The rotation section gives only the openbao/ path for encrypting OS-TLS-INT.pem. The other sections of this document also give the vault/ path and the environments/$KAYOBE_ENVIRONMENT/ paths. Users of Hashicorp Vault or environments who follow this section can leave the new intermediate CA key unencrypted in Git. Add the same path variants here.

The rotation playbook disables the OS-TLS-INT PKI secret engine on the seed and the controllers. The old issuer is unavailable until new certificates are generated and deployed. Add a warning in the same form as the one in "Certificates deployment".

Suggested fix

 Users can rotate the intermediate CA issuer with
 ``secret-store-rotate-intermediate-issuer.yml`` playbook.
 
+.. warning::
+
+   The rotation disables the ``OS-TLS-INT`` secret engine on the seed and the
+   controllers. The old issuer is unavailable until new certificates are
+   generated and deployed.
+
 .. code-block:: bash
 
    kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/secret-store/secret-store-rotate-intermediate-issuer.yml
 
 This will regenerate intermediate CA certificate files ``OS-TLS-INT.crt`` and
 ``OS-TLS-INT.pem``.
 Make sure to encrypt ``OS-TLS-INT.pem`` again.
 
 .. code-block:: bash
 
    ansible-vault encrypt --vault-password-file ~/vault.pass $KAYOBE_CONFIG_PATH/openbao/OS-TLS-INT.pem
+   ansible-vault encrypt --vault-password-file ~/vault.pass $KAYOBE_CONFIG_PATH/vault/OS-TLS-INT.pem
+   ansible-vault encrypt --vault-password-file ~/vault.pass $KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/openbao/OS-TLS-INT.pem
+   ansible-vault encrypt --vault-password-file ~/vault.pass $KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/vault/OS-TLS-INT.pem
Committable suggestion

IMPORTANT
Review the replacement and indentation, then test before committing.

Suggested change
Rotating intermediate CA issuer
-------------------------------
Users can rotate the intermediate CA issuer with
``secret-store-rotate-intermediate-issuer.yml`` playbook.
.. code-block:: bash
kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/secret-store/secret-store-rotate-intermediate-issuer.yml
This will regenerate intermediate CA certificate files ``OS-TLS-INT.crt`` and
``OS-TLS-INT.pem``.
Make sure to encrypt ``OS-TLS-INT.pem`` again.
.. code-block:: bash
ansible-vault encrypt --vault-password-file ~/vault.pass $KAYOBE_CONFIG_PATH/openbao/OS-TLS-INT.pem
Once new issuer is set, regenerate TLS certificates by following instructions
at :ref:`Certificates generation<certificates-generation>`, then deploy them
by following instructions at
:ref:`Certificates deployment<openbao-certificates>`.
Rotating intermediate CA issuer
-------------------------------
Users can rotate the intermediate CA issuer with
``secret-store-rotate-intermediate-issuer.yml`` playbook.
.. warning::
The rotation disables the ``OS-TLS-INT`` secret engine on the seed and the
controllers. The old issuer is unavailable until new certificates are
generated and deployed.
.. code-block:: bash
kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/secret-store/secret-store-rotate-intermediate-issuer.yml
This will regenerate intermediate CA certificate files ``OS-TLS-INT.crt`` and
``OS-TLS-INT.pem``.
Make sure to encrypt ``OS-TLS-INT.pem`` again.
.. code-block:: bash
ansible-vault encrypt --vault-password-file ~/vault.pass $KAYOBE_CONFIG_PATH/openbao/OS-TLS-INT.pem
ansible-vault encrypt --vault-password-file ~/vault.pass $KAYOBE_CONFIG_PATH/vault/OS-TLS-INT.pem
ansible-vault encrypt --vault-password-file ~/vault.pass $KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/openbao/OS-TLS-INT.pem
ansible-vault encrypt --vault-password-file ~/vault.pass $KAYOBE_CONFIG_PATH/environments/$KAYOBE_ENVIRONMENT/vault/OS-TLS-INT.pem
Once new issuer is set, regenerate TLS certificates by following instructions
at :ref:`Certificates generation<certificates-generation>`, then deploy them
by following instructions at
:ref:`Certificates deployment<openbao-certificates>`.

Comment on lines +7 to +9
If user's system already generated proxysql-ca.pem before, intermediate CA
can be rotated by using ``secret-store-rotate-intermediate-issuer.yml``
playbook.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major · Security & Privacy · Heavy lift

Document that rotation does not revoke the exposed intermediate CA.

The release note presents secret-store-rotate-intermediate-issuer.yml as the remediation for an existing proxysql-ca.pem. That playbook disables the OS-TLS-INT PKI engine. It does not revoke the previous intermediate certificate.

Certificates already signed with the exposed key remain valid until they expire. Controllers trust the root CA, so they can accept these certificates unless revocation is enforced. A holder of the leaked key can therefore issue certificates that controllers accept.

Update the release note to state this limit. Name the step that removes trust in the previous intermediate. If that step does not exist, add it to the rotation procedure.

The playbook ``secret-store-generate-internal-tls.yml`` has been
creating ``proxysql-ca.pem`` by copying ``OS-TLS-INT.pem``.
This file contains key of intermediate CA, resulting leaking the key to
controller hosts if ProxySQL is enabled.

Fixed by changing the source to ``OS-TLS-INT.crt`` which only has the
certificate of intermediate CA.

Added playbook and documentation for rotating intermediate CA issuer.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting-review PR is waiting for a review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant