[geekhubuser]

Description

The image no longer builds, on any PR or on master. On 2026-10-20 the Alpine 3.18 x86_64 index replaced the pinned ca-certificates 20241121-r1 with 20261020-r0, and Alpine keeps one version per branch:

#10 0.388 ERROR: unable to select packages:
#10 0.389   ca-certificates-20261020-r0:
#10 0.389     breaks: world[ca-certificates=20241121-r1]

The operator also cannot onboard a large repository. Since 2.29.0 go-git clones and pushes inside the operator process. With strategy clone, apache/kafka (312 MB) gets the operator OOM-killed at the default 1Gi limit while it pushes the branches and tags, on the current image and on a fixed one alike, and every restart starts the onboarding again, so the Codebase stays in progress.

This change:

• Runs Clone and Push with the git binary (pkg/git/cli.go, CLIProvider); every other operation stays on go-git. GIT_CLI_ENABLED (chart value gitCliEnabled, default true) set to false returns to go-git without a new image.
• Builds on alpine:3.24 with git~=2.54 and openssh-client~=10.3, pinned by minor as in gerrit-operator, and keeps master's entrypoint and user_setup, which give the random UID OpenShift assigns a passwd entry; ssh needs one.
• Keeps go-git's behaviour on the git path:
  • SSH host keys verified against SSH_KNOWN_HOSTS, with the same messages (sshhostkey.OpenSSHOptions, EnrichOutput);
  • the GitServer user (-l; Gerrit URLs carry none);
  • every SSH key go-git reads: the key is re-encoded through x/crypto, because ssh rejects spellings go-git accepts (CRLF line endings, surrounding blank lines, no final newline);
  • the token goes only to the remote's own scheme and host, through a credential helper git asks per host, so a redirect to another host or to plain http gets nothing; it travels in the environment, never in arguments;
  • the CAs Go trusts (SSL_CERT_FILE or the system bundle, plus the SSL_CERT_DIR directories), collected into GIT_SSL_CAINFO because git reads neither variable, without blocks that do not parse;
  • an atomic push (every ref or none); a server without atomic push gets a plain one;
  • an empty remote fails with ErrEmptyRemoteRepository, a remote HEAD that names a missing branch with ErrReferenceNotFound, and a push without refspecs pushes every branch;
  • no .gitattributes conversion on checkout, since go-git commits on that checkout later.
• Fixes for both engines:
  • Clone checks out after it has fetched every branch, so a push to the default branch during a long clone no longer leaves the working tree behind the branch, where the next commit would revert that push;
  • a failed or killed clone or fetch removes the partial clone, which the next reconcile would have taken for a complete one;
  • Checkout fetches branches and tags only, not refs/* (apache/kafka advertises about 29,000 pull request refs).
• Bounds git's memory: 2 delta threads (pack.threads), so it does not grow with the node's CPU count; a 100 MiB delta window per thread (pack.windowMemory); blobs over 64 MiB streamed without a delta search (core.bigFileThreshold).
• Passes git no inherited GIT_ variable except the config file locations, so GIT_DIR cannot point it at another repository and GIT_SSL_NO_VERIFY cannot turn TLS verification off; turns git maintenance off, because the operator is PID 1 and does not reap a detached process.
• At startup the operator logs which engine clones and pushes and any invalid GIT_CLI_ENABLED value, exits when the git binary is missing, and logs an error when ssh is missing or the UID has no passwd entry for it.

Fixes EPMDEDP-17426 (codebase-operator part; the gerrit-operator part is epam/edp-gerrit-operator#103)

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Enhancement (non-breaking change which improves an existing feature or documentation)
• Breaking change (fix or feature that would cause existing functionality not to work as expected)

How Has This Been Tested?

kind platform with GitLab (try-kuberocketci), codebase-operator switched to an image built from this branch, default 1Gi memory limit, cgroup v2:

Check	Result
apache/kafka (93 branches, 326 tags), strategy clone	created in 118 s; 0 restarts, oom_kill 0; every branch and tag in GitLab. With go-git the same onboarding was OOM-killed
kubernetes/kubernetes (1.5 GB, 62 branches, 1,251 tags), strategy clone	created in 510 s; 0 restarts, oom_kill 0; every branch and tag in GitLab
Memory at the 1Gi limit, sampled every 2 s	kafka: pod anon peak 244 MiB, git up to 544 MiB RSS, operator 196 MiB. kubernetes: anon 528 MiB, git up to 890 MiB RSS, operator 434 MiB. git's RSS includes the pack files it maps, which the kernel can reclaim; anon is what an OOM counts
10 versions of a 90 MB binary, git push under a 1 GiB limit in this image	with the thread cap only: anon 971 MiB, OOM kill, nothing pushed. With pack.windowMemory and core.bigFileThreshold: anon 87 MiB, all pushed
apache/kafka with defaultBranch: 4.0, not the source HEAD, so the go-git checkout fetch runs too	created; 0 restarts, oom_kill 0; every branch and tag in GitLab
strategy create (template over HTTPS, push over SSH), then a CodebaseBranch	created, both branches in GitLab; git and git-remote-http seen in the container
runAsUser: 1000680000, runAsGroup: 0, as OpenShift assigns, strategy clone	created; whoami resolves through the entrypoint; ssh ran
GIT_CLI_ENABLED=false	created with go-git; no git or ssh process in the container
Rollback to 2.37.0-SNAPSHOT.1	rolled out, 0 restarts

The kafka, kubernetes and memory rows ran on this commit, and the binary row ran this image's git with the operator's settings. The other rows ran on an earlier build of the branch, before the review changes (byte caps, cleanup after a failed clone).

Local checks, in golang:1.26 (Go 1.26.9) on a copy of the branch:

Check	Result
make build (amd64, arm64), make test	pass, 42 packages
make lint	0 issues
hadolint v2.12.0 (the pipeline's digest)	pass
helm lint, helm template	pass; GIT_CLI_ENABLED renders "true", and "false" with --set gitCliEnabled=false
make helm-docs	README unchanged
Image, amd64	git 2.54.0, OpenSSH 10.3p1; a random UID resolves only through the entrypoint
Image, arm64	git~=2.54 and openssh-client~=10.3 resolve from the aarch64 index; the image itself was not built here (no arm64 emulation on this host)
sshd in alpine:3.24 (OpenSSH 10.3)	clone and push with -l on a URL without a user; missing and changed host keys give the go-git messages; CRLF and no-newline keys work
Unit tests	a real HTTPS server (git http-backend, Basic auth, a CA from SSL_CERT_DIR), an ssh stand-in, a branch moved during the clone; each fix's test fails when the fix is removed

Not run live: the Gerrit, GitHub and Bitbucket providers and caCerts.enabled, because the testbed has GitLab only, and the import strategy. The sshd run covers the Gerrit URL form, the HTTPS unit test covers the CA path, and the import steps (clone, remote branch checkout, commit, push) were run against sshd outside a Codebase.

Checklist:

• I have performed a self-review of my code
• I have commented on my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Pull Request contains one commit. I squash my commits.

Additional context

Upgrade notes:

• Clone and push run the git binary by default; gitCliEnabled: false returns to go-git.
• git's memory counts toward resources.limits.memory like the operator's own. On cgroup v2 an OOM kills the whole container (memory.oom.group).
• git resolves deltas with 2 threads, so cloning a large repository can take longer on a node with many CPUs than an unbounded git would.
• ssh needs a passwd entry for the process UID. The image adds one for any UID in group 0, which is what OpenShift assigns. With a custom runAsUser and a non-zero runAsGroup, SSH clone and push fail, and the operator logs it at startup: use group 0, or gitCliEnabled: false.
• Proxies: git honours https_proxy/HTTPS_PROXY and no_proxy, but for http:// URLs only the lowercase http_proxy, and ssh ignores ALL_PROXY. go-git honoured both; if you rely on them, use gitCliEnabled: false.
• OpenSSH 10.3 refuses what go-git still accepted: DSA keys, a server that offers only SHA-1 key exchange, and a known_hosts entry written as [host]:22 (use the plain host form for port 22, as ssh-keyscan prints it).
• A proxy or load balancer in front of an HTTPS git server must accept chunked request bodies: git streams pushes larger than 1 MiB that way, where go-git sent a Content-Length.
• Clone and push write the SSH key and the CA bundle to a temporary directory for the call, so TMPDIR (/tmp) must be writable, also with a read-only root filesystem.
• The steps after a clone (checkout, commit) still run in go-git, which indexes every object of the pack in the operator's memory: the operator reached 434 MiB with kubernetes, so a repository with several times its object count can still hit the limit there.

Seen during the validation, not changed here: after the first push of a repository with many branches, GitLab's post-receive processing ran after the operator's default-branch call and kept the first pushed branch as the project default (kafka, kubernetes). This happens with go-git too.

Rollback: gitCliEnabled: false, with no image change. The previous image tag no longer builds, which is the bug.

#325 (EPMDEDP-17422) fails build-and-push for this reason and needs this change on master first.

[geekhubuser]

/ok-to-test

[epmd-edp]

Pipeline review-codebase-operator-master-mwj7q — ✓ Passed

Status	Task	Duration
✓ Passed	github-set-pending-status	11s
✓ Passed	fetch-repository	17s
✓ Passed	init-values	54s
✓ Passed	get-cache	2m46s
✓ Passed	commit-validate	48s
✓ Passed	dockerfile-lint	48s
✓ Passed	helm-lint	6s
✓ Passed	helm-docs	11s
✓ Passed	build	51s
✓ Passed	sonar	43s
✓ Passed	save-cache	2m21s
✓ Passed	buildkit-build	22s
✓ Passed	github-report-pipeline-status	6s

Pushing new commits re-runs this pipeline automatically.
To re-run it without new commits, comment /recheck.

[Oleksandr123234]

Question (non-blocking): this PR contains two changes: the base image fix (Alpine 3.18 -> 3.24, which unblocks every build, including #325) and switching clone/push from go-git to the git binary (~1.6k lines, mostly new code and tests).

Was there a reason to keep them together? Splitting them could let the build fix be reviewed and merged quickly, while the git CLI change gets a more focused review. If they depend on each other (e.g. the image needs git/ssh only because of the new code), that makes sense, I just wanted to ask.

[geekhubuser]

Thanks! This is intentional: the PO asked to bring the git CLI back in this PR, because go-git runs out of memory on large repositories.

[geekhubuser]

Closing: superseded by #327, which is on master as 791a2b2.