[salesforce] Bounded real-time batching for Login/Logout (requires Agent/Kibana 9.5) by shmsr · Pull Request #20932 · elastic/integrations

shmsr · 2026-08-27T08:06:54Z

Summary

Stacked on #20931 (base branch is salesforce-dedup-cursor-fixes). GitHub will retarget this PR to main automatically once #20931 merges. Review only the batching diff here; the dedup/cursor fixes are in #20931.

Adds bounded real-time collection for Login and Logout events so a large backfill can no longer issue an unbounded SOQL query:

New object.batch configuration on the Salesforce input: fixed-width, half-open (start, end] catch-up windows, up to max_windows_per_run per poll (default 12), window size defaulting to the real-time period.
Windowed object query using batch_start_time / batch_end_time (replaces the unbounded EventDate > first_event_time), ordered EventDate DESC.
New advanced settings: Real-Time Catch-Up Window (real_time_batch_window) and Max Real-Time Catch-Up Windows Per Poll (real_time_batch_max_windows).
Upgrade-safe: the 9.5 input seeds the first batched window from the legacy first_event_time watermark, so existing installs resume without replay or gaps.

Compatibility

object.batch requires the Agent 9.5 Salesforce input, so this bumps the package floor to kibana: ^9.5.0 + agent: ^9.5.0 and releases as 1.10.0. Sub-9.5 users continue to receive 1.9.0 (from #20931).

Test plan

elastic-package check / lint / build
elastic-package test system on a 9.5.0 stack: login 12/12 configs pass (incl. catchup, catchup-capped, window-size, independent-window, pagination, multi-record, sparse/empty windows, window/pagination failures), logout 7/7 pass.

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

mergify · 2026-08-27T08:08:29Z

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

Queue this pull request

Copilot

Pull request overview

This PR updates the Salesforce integration to use Elastic Agent 9.5’s object.batch support to collect real-time LoginEvent/LogoutEvent data in bounded (start, end] windows, preventing a large initial backfill from issuing open-ended SOQL queries. It also raises the package compatibility floor to Kibana/Agent 9.5 and adds system-test and mock updates to exercise windowing, caps, empty windows, pagination, and failure handling.

Changes:

Bump package to 1.10.0 and require Kibana 9.5.0 + Elastic Agent 9.5.0.
Add real-time batching configuration (object.batch) and new advanced tuning vars for window size and per-poll window caps.
Update docs, docker mock rules, and add system tests covering windowing/catch-up behaviors for login/logout real-time objects.

Reviewed changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
packages/salesforce/manifest.yml	Bumps version to 1.10.0 and raises Kibana/Agent minimums to 9.5.0 for `object.batch`.
packages/salesforce/docs/README.md	Documents 9.5 requirement and explains bounded real-time catch-up + scaling guidance.
packages/salesforce/_dev/build/docs/README.md	Mirrors rendered-doc source updates for batching + scaling guidance.
packages/salesforce/changelog.yml	Adds 1.10.0 changelog entry describing bounded real-time batching and new requirements.
packages/salesforce/data_stream/login/manifest.yml	Adds advanced vars for catch-up window + max windows; clarifies real-time polling semantics.
packages/salesforce/data_stream/login/agent/stream/salesforce.yml.hbs	Enables `object.batch` and switches object query to `(batch_start_time, batch_end_time]` windows.
packages/salesforce/data_stream/login/_dev/test/system/test-object-config.yml	Validates empty initial interval behavior for batched login object collection.
packages/salesforce/data_stream/login/_dev/test/system/test-object-catchup-config.yml	Tests draining a 1h lookback via 12 bounded windows (login object).
packages/salesforce/data_stream/login/_dev/test/system/test-object-catchup-capped-config.yml	Tests max-windows cap limiting per-poll catch-up (login object).
packages/salesforce/data_stream/login/_dev/test/system/test-object-window-size-config.yml	Tests window width following real_time_period (login object).
packages/salesforce/data_stream/login/_dev/test/system/test-object-independent-window-config.yml	Tests window override independent of poll interval (login object).
packages/salesforce/data_stream/login/_dev/test/system/test-object-empty-windows-config.yml	Tests that empty windows don’t halt catch-up (login object).
packages/salesforce/data_stream/login/_dev/test/system/test-object-sparse-windows-config.yml	Tests sparse windows don’t skip later events (login object).
packages/salesforce/data_stream/login/_dev/test/system/test-object-window-failure-config.yml	Tests window failure handling doesn’t skip successful window ingestion (login object).
packages/salesforce/data_stream/login/_dev/test/system/test-object-pagination-config.yml	Tests pagination within a bounded window (login object).
packages/salesforce/data_stream/login/_dev/test/system/test-object-pagination-failure-config.yml	Tests pagination failure handling within a bounded window (login object).
packages/salesforce/data_stream/login/_dev/test/system/test-object-multi-record-config.yml	Tests multi-record results within a bounded window (login object).
packages/salesforce/data_stream/logout/manifest.yml	Adds advanced vars for catch-up window + max windows; clarifies real-time polling semantics.
packages/salesforce/data_stream/logout/agent/stream/salesforce.yml.hbs	Enables `object.batch` and switches object query to `(batch_start_time, batch_end_time]` windows.
packages/salesforce/data_stream/logout/_dev/test/system/test-object-config.yml	Validates empty initial interval behavior for batched logout object collection.
packages/salesforce/data_stream/logout/_dev/test/system/test-object-catchup-config.yml	Tests draining a 1h lookback via 12 bounded windows (logout object).
packages/salesforce/data_stream/logout/_dev/test/system/test-object-catchup-capped-config.yml	Tests max-windows cap limiting per-poll catch-up (logout object).
packages/salesforce/data_stream/logout/_dev/test/system/test-object-window-size-config.yml	Tests window width following real_time_period (logout object).
packages/salesforce/data_stream/logout/_dev/test/system/test-object-empty-windows-config.yml	Tests that empty windows don’t halt catch-up (logout object).
packages/salesforce/data_stream/logout/_dev/test/system/test-object-window-failure-config.yml	Tests window failure handling doesn’t skip successful window ingestion (logout object).
packages/salesforce/_dev/deploy/docker/files/config.yml	Updates mock matching rules to accept batched LoginEvent/LogoutEvent SOQL window queries and sequences.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated no new comments.

Suppressed comments (2)

packages/salesforce/data_stream/login/agent/stream/salesforce.yml.hbs:41

query.default becomes unbounded when initial_interval is an empty string (the {{#if initial_interval}} block is skipped), which can result in a full-table SOQL query on first run. Since the intent here is to keep real-time object queries bounded, always include a lower bound, falling back to real_time_period when initial_interval is empty.

      default: SELECT FIELDS(STANDARD) FROM LoginEvent{{#if initial_interval}} WHERE EventDate > [[ (formatTime (now.Add (parseDuration "-{{initial_interval}}")) "2006-01-02T15:04:05.000Z0700") ]]{{/if}} ORDER BY EventDate DESC

packages/salesforce/data_stream/logout/agent/stream/salesforce.yml.hbs:41

query.default is unbounded when initial_interval is an empty string (the conditional WHERE clause is omitted), which can lead to a full-table SOQL query. To keep real-time object collection bounded, always add a lower bound and fall back to real_time_period when initial_interval is empty.

      default: SELECT FIELDS(STANDARD) FROM LogoutEvent{{#if initial_interval}} WHERE EventDate > [[ (formatTime (now.Add (parseDuration "-{{initial_interval}}")) "2006-01-02T15:04:05.000Z0700") ]]{{/if}} ORDER BY EventDate DESC

Copilot

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated no new comments.

Collect real-time Login and Logout events in fixed-width, bounded time windows (up to a max number of windows per poll) so a large backfill can no longer issue an unbounded SOQL query. Adds advanced Real-Time Catch-Up Window and Max Windows Per Poll settings. This uses object.batch on the Salesforce input, which requires Elastic Agent 9.5.0+, so the package now requires Kibana ^9.5.0 and Agent ^9.5.0. Stacked on top of the dedup/cursor fixes (1.9.0), which keep the lower compatibility range.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Address review: when initial_interval is empty the {{#if}} block was skipped, leaving query.default as an unbounded LoginEvent/LogoutEvent scan. object.batch drives execution via query.value, but bound the default too as defense-in-depth, falling back to real_time_period when initial_interval is empty (mirroring batch.initial_interval).

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

elastic-vault-github-plugin-prod · 2026-08-27T09:12:03Z

✅ All changelog entries have the correct PR link.

elastic-vault-github-plugin-prod · 2026-08-27T10:18:55Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

infra-vault-gh-plugin-prod · 2026-08-27T10:18:59Z

💚 Build Succeeded

Buildkite Build
Commit: 2615a54

cc @shmsr

Copilot AI lite review requested due to automatic review settings August 27, 2026 08:06

shmsr requested a review from a team as a code owner August 27, 2026 08:06

shmsr added a commit that referenced this pull request Aug 27, 2026

[salesforce] Point 1.10.0 changelog link at PR #20932

80b23c2

shmsr mentioned this pull request Aug 27, 2026

[salesforce] Bounded real-time batching and same-timestamp resume #20884

Closed

14 tasks

Copilot started reviewing on behalf of shmsr August 27, 2026 08:07 View session

shmsr added a commit that referenced this pull request Aug 27, 2026

[salesforce] Point 1.10.0 changelog link at PR #20932

ffa3083

shmsr force-pushed the salesforce-realtime-batching branch from 80b23c2 to ffa3083 Compare August 27, 2026 08:10

Copilot AI reviewed Aug 27, 2026

View reviewed changes

Comment thread packages/salesforce/data_stream/login/agent/stream/salesforce.yml.hbs Outdated

Comment thread packages/salesforce/data_stream/logout/agent/stream/salesforce.yml.hbs Outdated

Copilot AI review requested due to automatic review settings August 27, 2026 08:11

shmsr self-assigned this Aug 27, 2026

shmsr added enhancement New feature or request Integration:salesforce Salesforce Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] labels Aug 27, 2026

Copilot started reviewing on behalf of shmsr August 27, 2026 08:12 View session

Copilot AI reviewed Aug 27, 2026

View reviewed changes

shmsr added a commit to shmsr/integrations that referenced this pull request Aug 27, 2026

[salesforce] Point 1.10.0 changelog link at PR elastic#20932

c11a5e2

shmsr added a commit that referenced this pull request Aug 27, 2026

[salesforce] Point 1.10.0 changelog link at PR #20932

f88b73a

Copilot AI review requested due to automatic review settings August 27, 2026 08:19

shmsr force-pushed the salesforce-realtime-batching branch from ffa3083 to f88b73a Compare August 27, 2026 08:19

Copilot started reviewing on behalf of shmsr August 27, 2026 08:20 View session

Copilot AI reviewed Aug 27, 2026

View reviewed changes

shmsr added a commit that referenced this pull request Aug 27, 2026

[salesforce] Point 1.10.0 changelog link at PR #20932

5bda949

Copilot AI review requested due to automatic review settings August 27, 2026 08:37

shmsr force-pushed the salesforce-realtime-batching branch from f88b73a to 5bda949 Compare August 27, 2026 08:37

Copilot started reviewing on behalf of shmsr August 27, 2026 08:38 View session

Copilot AI reviewed Aug 27, 2026

View reviewed changes

[salesforce] Point 1.10.0 changelog link at PR #20932

ddd8083

Copilot AI review requested due to automatic review settings August 27, 2026 08:44

shmsr force-pushed the salesforce-realtime-batching branch from 5bda949 to ddd8083 Compare August 27, 2026 08:44

Copilot AI reviewed Aug 27, 2026

Copilot AI review requested due to automatic review settings August 27, 2026 09:06

Copilot AI reviewed Aug 27, 2026

Navigation Menu

Uh oh!

[salesforce] Bounded real-time batching for Login/Logout (requires Agent/Kibana 9.5) - #20932

[salesforce] Bounded real-time batching for Login/Logout (requires Agent/Kibana 9.5)#20932
shmsr wants to merge 3 commits into
salesforce-dedup-cursor-fixesfrom
salesforce-realtime-batching

shmsr commented Aug 27, 2026

Uh oh!

github-actions Bot commented Aug 27, 2026

Uh oh!

mergify Bot commented Aug 27, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 27, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 27, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Aug 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

shmsr commented Aug 27, 2026

Summary

Compatibility

Test plan

Related

Uh oh!

github-actions Bot commented Aug 27, 2026

✅ Elastic Docs Style Checker (Vale)

Uh oh!

mergify Bot commented Aug 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 27, 2026

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 27, 2026

🚀 Benchmarks report

Uh oh!

infra-vault-gh-plugin-prod Bot commented Aug 27, 2026

💚 Build Succeeded

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants