Skip to content

CI: Take the report merge off the test fan-out, stop the agent pod churn, report split balance - #5104

Open
michaelsembwever wants to merge 1 commit into
apache:trunkfrom
thelastpickle:mck/ci-perf/trunk
Open

CI: Take the report merge off the test fan-out, stop the agent pod churn, report split balance#5104
michaelsembwever wants to merge 1 commit into
apache:trunkfrom
thelastpickle:mck/ci-perf/trunk

Conversation

@michaelsembwever

Copy link
Copy Markdown
Member

Three faults measured on a Kubernetes clone of this pipeline.

Jenkinsfile:

jenkins-deployment.yaml:

  • slaveConnectTimeout 600 and waitForPodSec 900, the two deadlines on one provisioning attempt. 30 seconds met only a warm node, and churned 1,110 pods in one 24 minute window with 0 agents connected and 483 KubernetesProvisioningLimits ... went below zero warnings
  • idleMinutes 5, a pod holding a whole node
  • the four instanceCaps to the demand a pre-commit run measured; containerCap to their sum, so no pool's share is decided by which holds its agents longest, and maxRequestsPerHostStr with it
  • add the agent-dind-report template, on a node pool of its own

cell_balance.py:

  • new. Per target, how evenly its splits divided and what the worst cell ran, to the console and ci_summary.html. Splits stay round-robin, so the lever is the split count
  • guarded in generate-ci-summary.sh, never failing a build

run-ci-test.py:

  • the fixture's node group maximums follow the raised caps

Comment thread .jenkins/Jenkinsfile Outdated
Comment thread .jenkins/Jenkinsfile Outdated
Comment thread .jenkins/Jenkinsfile
Comment thread .build/ci/cell_balance.py Outdated
Comment thread .jenkins/Jenkinsfile Outdated
Comment thread .jenkins/Jenkinsfile Outdated
Comment thread .jenkins/Jenkinsfile
…urn, report split balance

Three faults measured on a Kubernetes clone of this pipeline.

 Jenkinsfile:
  - generateTestReports moves from cassandra-medium, the label of the ~460 test splits it queued behind, to cassandra-report: 109 minutes of a 163 minute build, for ~3 minutes of work
  - one ant junitreport per target, three at a time under an explicit -Xmx
  - size every `xargs -P` and `xz -T` from the container's cgroup quota.  nproc answers for the node: 8 against a 2 cpu limit
  - record each cell's duration and longest suites, and archive them
  - test-burn takes 5 splits, which separates its three long classes
  - remove testDataPublishers as they were oom-ing ci-cassandra.a.o's controller https://the-asf.slack.com/archives/CK23JSY2K/p1787841152008959?thread_ts=1787836602.176469&cid=CK23JSY2K , and it was noise because of jenkinsci/test-stability-plugin#11 (we use butler instead)
  - three helpers out of test(..) for readability: recordCellTime, processResults, testScriptVars

 jenkins-deployment.yaml:
  - slaveConnectTimeout 600 and waitForPodSec 900, the two deadlines on one provisioning attempt.  30 seconds met only a warm node, and churned 1,110 pods in one 24 minute window with 0 agents connected and 483 `KubernetesProvisioningLimits ... went below zero` warnings
  - idleMinutes 5, a pod holding a whole node
  - the four instanceCaps to the demand a pre-commit run measured; containerCap to their sum, so no pool's share is decided by which holds its agents longest, and maxRequestsPerHostStr with it
  - add the agent-dind-report template, on a node pool of its own

 cell_balance.py:
  - new.  Per target, how evenly its splits divided and what the worst cell ran, to the console and ci_summary.html.  Splits stay round-robin, so the lever is the split count
  - guarded in generate-ci-summary.sh, never failing a build

 run-ci-test.py:
  - the fixture's node group maximums follow the raised caps

 patch by Mick Semb Wever; reviewed by Brandon Williams, Dmitry Konstantinov for CASSANDRA-21640

Assisted-by: Claude Code:claude-opus-5
@michaelsembwever

Copy link
Copy Markdown
Member Author

squashed and message written.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants