Skip to content

EPMDEDP-17426: fix: Cap git pack memory, push atomically, pin apk minors - #328

Merged
SergK merged 1 commit into
masterfrom
EPMDEDP-17426
Nov 2, 2026
Merged

SergK merged 1 commit into
masterfrom
EPMDEDP-17426

Conversation

@geekhubuser

Copy link
Copy Markdown
Member

Description

Follow-up to #327 (791a2b2), which moved clone, push, checkout and commit to the git binary. Three gaps remain:

  • A push of large files still exceeds the memory limit. git's delta search keeps up to 10 whole objects in memory with no byte limit, because pack.windowMemory is unset. Ten commits of a 90 MB file, pushed with the git settings of 791a2b2 at a 1 GiB limit, are OOM-killed and push nothing.
  • A partial push is adopted as complete. A push without --atomic can update some refs while the remote rejects others. On the next reconcile adoptPushedProject finds the default branch on the remote and adopts the push, so the rejected branches and tags never reach the remote.
  • The exact apk pins break the build again at the next Alpine update. Alpine keeps only the latest build of each package per branch; this is how ca-certificates=20241121-r1 broke every build on 2026-10-20.

This change:

  • Adds pack.windowMemory=100m and core.bigFileThreshold=64m to the git config of every call: the delta search holds at most 100 MiB of objects, and blobs over 64 MiB skip it and are streamed.
  • Pushes with --atomic, so a rejected ref leaves every ref unchanged and the next reconcile pushes again. A remote that does not offer atomic push (the receiving end does not support --atomic push) gets a plain push.
  • Installs git~=2.49 and openssh-client~=10.0, the minors alpine 3.22 ships, after apk upgrade, and takes the CA bundle from the base image's ca-certificates-bundle instead of the pinned ca-certificates package. Nothing in the image or the chart runs update-ca-certificates; custom CAs still come through SSL_CERT_DIR.

Jira: EPMDEDP-17426

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Enhancement (non-breaking change which improves an existing feature or documentation)
  • Breaking change (fix or feature that would cause existing functionality not to work as expected)

How Has This Been Tested?

Unit and lint checks in golang:1.26 (git 2.47.3), on a copy of the change:

Check Result
go test ./pkg/git/ pass, with the new TestCLIGitProvider_Push_Atomic (a rejected dev holds back main, git reports atomic push failed) and TestCLIGitProvider_Push_RemoteWithoutAtomic (receive.advertiseAtomic=false; branches and a tag pushed, as onboarding does)
The change reverted one part at a time push without --atomic: Push_Atomic fails; no fallback: Push_RemoteWithoutAtomic fails; no byte caps: TestBaseEnv_Config fails
make build (amd64, arm64), make test pass, 42 packages; pkg/git covers 90.7% of statements
make lint 0 issues
helm lint, helm template, make helm-docs pass, no drift
hadolint v2.12.0 pass

A push of 10 commits of a 90 MB random file (859 MB) to a bare repository, in alpine:3.22.7 with git 2.49.1-r0, at --memory=1g:

git settings Result
As in 791a2b2: gc.auto=0, maintenance.auto=false, pack.threads=1 OOM-killed after 28 s, anon peak 970 MiB, 0 commits pushed
Plus pack.windowMemory=100m, core.bigFileThreshold=64m pass in 34 s, anon peak 87 MiB, 10 commits pushed

The amd64 image of this commit: git 2.49.1, OpenSSH 10.0p2, ca-certificates-bundle-20261020-r0 (118 certificates), no package left to upgrade; git ls-remote of a GitHub repository over HTTPS succeeds with that bundle, and the entrypoint still gives a random UID in group 0 its passwd entry. In the v3.22 aarch64 index git 2.49.1-r0 and openssh-client 10.0_p1-r10 (provided by openssh-client-default) match the pins.

On a kind cluster running KubeRocketCI from main with GitLab, the operator image was swapped for an image of this commit and back:

Case Result
Strategy clone of go-git/go-git (45 branches, 98 tags) over HTTPS, push to GitLab over SSH created; GitLab has all 45 branches and 98 tags; the push was atomic, no fallback
Strategy create, then a CodebaseBranch both created; GitLab has main and the new branch
Back to 2.37.0-SNAPSHOT.1 rolled out, 0 restarts

Checklist:

  • I have performed a self-review of my code
  • I have commented on my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Pull Request contains one commit. I squash my commits.

Additional context

  • The fallback matches git's English message; the provider runs git with LC_ALL=C.
  • PutDeployConfigs and PutGitLabCIConfig push every local branch. When the remote rejects one, their commit to the default branch is now held back with it; before, it landed while the step still failed.
  • core.bigFileThreshold also makes git treat files over 64 MiB as binary, so it applies no end-of-line conversion to them.
  • The ~= pins follow the minors of the FROM tag: bump them together.
  • Rollback: revert the commit. No CRD or chart value changes.

- Cap the delta search window at 100 MiB and stream blobs over 64 MiB, so a push of large files stays under the memory limit
- Push with --atomic, so a rejected ref leaves the remote unchanged; a remote without atomic push gets a plain push
- Pin git and openssh-client by minor and upgrade the base packages, so the next Alpine package update does not break the build
- Take the CA bundle from the base image instead of a pinned ca-certificates package
@geekhubuser geekhubuser self-assigned this Nov 2, 2026
@geekhubuser
geekhubuser requested a review from a team as a code owner November 2, 2026 10:41
@geekhubuser
geekhubuser requested a review from SergK November 2, 2026 10:42
@epmd-edp

epmd-edp commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator

Pipeline review-codebase-operator-master-rg77s — ✓ Passed

Status Task Duration
✓ Passed github-set-pending-status 7s
✓ Passed fetch-repository 13s
✓ Passed init-values 5s
✓ Passed get-cache 1m56s
✓ Passed commit-validate 5s
✓ Passed dockerfile-lint 7s
✓ Passed helm-lint 6s
✓ Passed helm-docs 40s
✓ Passed build 47s
✓ Passed sonar 17s
✓ Passed save-cache 8s
✓ Passed buildkit-build 20s
✓ Passed github-report-pipeline-status 7s

Pushing new commits re-runs this pipeline automatically.
To re-run it without new commits, comment /recheck.

@SergK
SergK merged commit 4233c5c into master Nov 2, 2026
6 checks passed
@SergK
SergK deleted the EPMDEDP-17426 branch November 2, 2026 11:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants