Skip to content

.buildkite/scripts: derive affected package list from git diff - #20926

Open
efd6 wants to merge 1 commit into
elastic:mainfrom
efd6:check_changed_packages
Open

.buildkite/scripts: derive affected package list from git diff#20926
efd6 wants to merge 1 commit into
elastic:mainfrom
efd6:check_changed_packages

Conversation

@efd6

@efd6 efd6 commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Proposed commit message

.buildkite/scripts: derive affected package list from git diff

Replace the linear scan of all packages with a diff-derived candidate
list. Before the loop, compute the merge base and changed files once,
then extract affected package paths with grep. Only iterate those
packages instead of all 486.

Fall back to the full list_all_directories scan when non-package files
are changed or FORCE_CHECK_ALL is set. Export COMMIT_MERGE so
is_pr_affected can reuse it instead of re-running git merge-base for
each candidate package.

Applied to both trigger_integrations_in_parallel.sh and
test_integrations_with_serverless.sh.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

@efd6 efd6 self-assigned this Aug 27, 2026
@efd6 efd6 added enhancement New feature or request Team:Ecosystem Packages Ecosystem team [elastic/ecosystem] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Aug 27, 2026
@elastic-vault-github-plugin-prod

elastic-vault-github-plugin-prod Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

🚀 Benchmarks report

Package windows 👍(5) 💚(4) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
powershell_operational 2808.99 1674.48 -1134.51 (-40.39%) 💔

To see the full report comment with /test benchmark fullreport

@efd6

efd6 commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

/test

1 similar comment
@efd6

efd6 commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

/test

@efd6
efd6 marked this pull request as ready for review August 27, 2026 04:19
@efd6
efd6 requested a review from a team as a code owner August 27, 2026 04:19
@infra-vault-gh-plugin-prod

Copy link
Copy Markdown

Pinging @elastic/ecosystem (Team:Ecosystem)

@infra-vault-gh-plugin-prod

Copy link
Copy Markdown

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@efd6

efd6 commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

The test failures here are due to infrastructure flakiness.

Comment thread .buildkite/scripts/test_integrations_with_serverless.sh
Comment thread .buildkite/scripts/trigger_integrations_in_parallel.sh
Replace the linear scan of all packages with a diff-derived candidate
list. Before the loop, compute the merge base and changed files once,
then extract affected package paths with grep. Only iterate those
packages instead of all 486.

Fall back to the full list_all_directories scan when non-package files
are changed or FORCE_CHECK_ALL is set. Export COMMIT_MERGE so
is_pr_affected can reuse it instead of re-running git merge-base for
each candidate package.

Applied to both trigger_integrations_in_parallel.sh and
test_integrations_with_serverless.sh.
@efd6
efd6 force-pushed the check_changed_packages branch from 7d9cde0 to 66e1507 Compare August 27, 2026 04:33
@vera-review-bot

Copy link
Copy Markdown

🟢 Reviewed the latest commits 66e1507 — nothing new beyond already posted comments.

Review summary

Issues found across the latest commits 7d9cde0 — 2 critical
  • 🔴 The affected-package computation aborts the build under set -euo pipefail: when the second grep finds no match its non-zero exit propagates through the pipe and fails the PACKAGE_LIST=$(...) assignment. Make the greps non-fatal with || true. (link) (Unresolved)
  • 🔴 Same set -euo pipefail abort in the dynamic-pipeline generator: the second grep's no-match exit fails the PACKAGE_LIST=$(...) assignment, so no test steps get uploaded. Guard the greps with || true. (link) (Unresolved)

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot - v0.2.7 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

@teresaromero teresaromero left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor comment. thanks!!

changed_files=$(git diff --name-only "${COMMIT_MERGE}" "${to}")

if [[ "${FORCE_CHECK_ALL}" == "true" ]] || echo "${changed_files}" | pr_has_package_related_files; then
echo "Non-package files changed or force-check enabled: scanning all packages"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
echo "Non-package files changed or force-check enabled: scanning all packages"
echo "Non-package files changed or FORCE_CHECK_ALL set: scanning all packages"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Team:Ecosystem Packages Ecosystem team [elastic/ecosystem] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants