Skip to content

ci: reduce SCA memory pressure on ARM runners - #472

Merged
XuPeng-SH merged 1 commit into
mainfrom
codex/issue31428-sca-memory-limit
Oct 28, 2026
Merged

XuPeng-SH merged 1 commit into
mainfrom
codex/issue31428-sca-memory-limit

Conversation

@XuPeng-SH

Copy link
Copy Markdown
Contributor

Problem

The default ARM SCA runner has an 8 GiB cgroup limit. On the full-scan workload from matrixorigin/matrixone#31428, the control run's long-lived golangci-lint process reached 7.20 GiB RSS; the cgroup had 7.30 GiB of anonymous memory and only about 7 MiB of inactive file cache at that point. The original failed job's log is unavailable, so the process killed in that incident is not established.

Change

Set a default GOMEMLIMIT=5GiB on the existing modern make static-check-analysis invocation, retaining any explicitly supplied value. This is a soft target per Go process, not a cgroup memory limit. It reaches inventory, vet, compiler children and golangci. Package selection, enabled linters and build parallelism are unchanged. The legacy make static-check path remains unchanged.

Related to matrixorigin/matrixone#31428. Keep the issue open: this is a measured mitigation of the analysis peak, not proof that the original incident or every whole-job OOM is eliminated.

Validation

Ran sequential full SCA controls on the ARM 4c/8 GiB runner class using the same incident source, full package graph, Go and linter versions and observed cache restore keys: baseline, candidate. Both passed and recorded no OOM or OOM-kill events.

Measurement Baseline Candidate
Golangci analysis peak RSS 7.202 GiB 5.174 GiB
Cgroup anonymous peak during analysis 7.303 GiB 5.245 GiB
make static-check-analysis elapsed 35m 11.746s 34m 31.077s
Make user + system CPU 6068.783s 5977.027s

The full package graph and 12 enabled golangci linters were unchanged; both had zero final golangci findings and the 139 emitted molint diagnostic records matched. This establishes output parity, not correctness of the existing molint failure gating, which is tracked separately. These are single-run measurements; effective CPU quota, host load and exact build/page-cache warmth were not recorded, so they do not predict a precise timing improvement.

  • The actual workflow shell passed checks for default and explicit limits, off, existing GOFLAGS, legacy handling and license/analysis failures.
  • A real Go 1.27.1 Make-to-runtime fixture verified inheritance of the default, an explicit lower value and off.
  • actionlint and git diff --check passed. There is no Go source or test change; Go UT race and BVT are not applicable to this workflow-only change.
  • The exact diff and evidence were reviewed independently with gpt-6.1-terra at max reasoning effort.

Remaining risk

The candidate's loader reached a sampled current - inactive_file peak of 7.919 GiB, versus 7.358 GiB in the baseline. This misses the proposed 7.5 GiB whole-scan headroom target; the metric includes file memory that may be reclaimable and is not a hard measure of unreclaimable memory. The candidate's loader had four memory.events max increments versus none in the baseline loader; one pair cannot determine causation. The Go soft limit does not bound aggregate compiler or non-Go memory. Further evidence is needed if loader OOM recurs; this change does not guarantee that the whole job stays below the container limit.

The legacy 4.2 path and SCA on larger runner overrides have not been benchmarked with this setting. An explicit higher limit or off can remove the mitigation.

@XuPeng-SH
XuPeng-SH merged commit 79c604b into main Oct 28, 2026
1 check passed
@XuPeng-SH
XuPeng-SH deleted the codex/issue31428-sca-memory-limit branch October 28, 2026 05:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant