ci-tools: bump bin build memory request to 3Gi - #83341
Conversation
The bin-arm64 image build (go install of ~50 cmd/ binaries plus the pod-scaler/repo-init webpack bundles) only requested the repo-wide default of cpu:1/memory:1Gi, same as every other lightweight step. bin-amd64 needs ~23 minutes to complete this work; bin-arm64 has been observed failing consistently around the 7 minute mark with DockerBuildFailed, consistent with the build getting resource-starved under the 1Gi request rather than hitting an actual compile error. Reproduced building the same commit locally on arm64 hardware with no compile errors, and openshift/ci-tools#5363 (unrelated diff, same base SHA) showed the same failure -- ruling out a code regression. coredns already carries an identical override for the same reason (ci-operator/config/openshift/coredns/openshift-coredns-main.yaml). Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository YAML (base), Central YAML (inherited) Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
WalkthroughThe wildcard resource configuration now requests 1 CPU and 3 GiB of memory for the ChangesResource profile configuration
Estimated code review effort: 1 (Trivial) | ~2 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 15✅ Passed checks (15 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
/pj-rehearse pull-ci-openshift-ci-tools-main-images Note Responses generated with Claude |
|
@kaovilai: your |
|
/pj-rehearse pull-ci-openshift-ci-tools-main-images Note Responses generated with Claude |
|
@kaovilai: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
[REHEARSALNOTIFIER]
Prior to this PR being merged, you will need to either run and acknowledge or opt to skip these rehearsals. Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
Rehearsal failure was infra congestion, not a repeat of the arm64 memory issue — /pj-rehearse pull-ci-openshift-ci-tools-main-images Note Responses generated with Claude |
|
@kaovilai: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
Rehearsal passed clean on retry, on this PR's current head commit ( /pj-rehearse ack Note Responses generated with Claude |
|
@kaovilai: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
openshift-merge-bot
Bot
added
the
rehearsals-ack
|
/approve |
1 similar comment
|
/approve |
openshift-ci
Bot
added
the
approved
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: bear-redhat, bearnl, kaovilai, shubham-pampattiwar, sseago, weshayutin The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@kaovilai: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
6 participants
What
ci-operator/config/openshift/ci-tools/openshift-ci-tools-main.yaml'sbinbuild step (go installof ~50cmd/binaries plus the pod-scaler/repo-init webpack bundles) was only getting the repo-wide defaultcpu:1/memory:1Gi, same as every other lightweight step. Added a dedicatedbin:override atmemory: 3Gi(keepingcpu: "1").Why
ci/prow/imageson openshift/ci-tools#5363 failed 5/5 times, always at thebin-arm64build step withDockerBuildFailedaround the 7-minute mark, whilebin-amd64needs ~23 minutes to complete the same work and succeeds.Ruled out a code regression:
GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go build ./...and the fullmake production-install) with no compile errors.imagesjob passed clean.This is consistent with the arm64 build getting resource-starved under the 1Gi request rather than hitting an actual error.
corednsalready carries an identical override for the same reason (ci-operator/config/openshift/coredns/openshift-coredns-main.yaml).No
make jobs/prowgen regen needed — confirmedci-operator/jobs/openshift/ci-tools/*.yamldoesn't bake in resource requests; theresources:block is read by ci-operator at runtime.Test plan
/pj-rehearsethis PR and confirmpull-ci-openshift-ci-tools-main-images(or a rehearsal of it) runs with the bumped memory and completesbin-arm64successfully.Summary by CodeRabbit
The OpenShift CI build for
openshift-ci-toolsnow requests 3 GiB of memory and 1 CPU for thebinstep. This prevents consistent arm64 build failures caused by memory starvation while preserving the existing CPU allocation.