[security_ai_prompts] Sync prompt changes from elastic/kibana#271330 by NicholasPeretti · Pull Request #20880 · elastic/integrations

NicholasPeretti · 2026-08-24T16:14:54Z

Summary

Syncs Security AI prompt saved objects from elastic/kibana#271330, which trims ALERT_SUMMARY_500 and ALERT_SUMMARY_SYSTEM_PROMPT to remove redundant instructions while preserving their behavioural contracts.

Replaces #19388, which was closed by the stale bot and cannot be reopened. This branch is the same change rebased onto current main.

Changes

Updates the default prompt text of 4 existing saved objects — no files added, removed, or renamed:

Saved object	`promptId`	`promptGroupId`
`security_ai_prompts-9da0cebb-…`	`alertSummary`	`aiForSoc`
`security_ai_prompts-7d7cf0c3-…`	`alertSummary`	`ease`
`security_ai_prompts-26eb85d3-…`	`alertSummarySystemPrompt`	`aiForSoc`
`security_ai_prompts-d8aebe8c-…`	`alertSummarySystemPrompt`	`ease`

Plus the package version bump 1.0.13 → 1.0.14 and a changelog entry.

alertSummarySystemPrompt keeps the { summary, recommendedActions } JSON contract verbatim — the EASE alert summary flyout depends on JSON.parse succeeding against it, and silently falls back to raw text (dropping recommendedActions) if it fails. alertSummary keeps the 500/200 character constraints.

All 4 values were verified byte-for-byte against the Kibana PR head, so the two repos stay in sync.

Test plan

elastic-package lint passes in packages/security_ai_prompts/
Saved objects install correctly after reinstalling the Security AI Prompts integration
EASE alert summary flyout renders both summary and recommendedActions (i.e. JSON.parse succeeds)
aiForSoc alert summary respects the 500/200 character limits

🤖 Generated with Claude Code

Updates ALERT_SUMMARY_500 (alertSummary) and ALERT_SUMMARY_SYSTEM_PROMPT (alertSummarySystemPrompt) prompt content for both aiForSoc and ease promptGroupIds. Bumps version to 1.0.14. UUID-named files are preserved; no file renaming. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

elastic#19388 was closed by the stale bot and cannot be reopened, so the changelog link now references the replacement PR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

github-actions · 2026-08-24T16:16:24Z

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

Copilot

Pull request overview

Syncs the Security AI Prompts integration package with the updated alert summary prompt text from elastic/kibana#271330, reducing redundancy while preserving the existing output contracts (500/200 char limits for alertSummary, and the { summary, recommendedActions } JSON contract for alertSummarySystemPrompt).

Changes:

Bump security_ai_prompts package version to 1.0.14.
Update alertSummary prompt text for both aiForSoc and ease prompt groups.
Update alertSummarySystemPrompt instructions for both aiForSoc and ease prompt groups, keeping the single-line JSON.parse()-compatible output requirement.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
packages/security_ai_prompts/manifest.yml	Version bump to `1.0.14`.
packages/security_ai_prompts/changelog.yml	Add `1.0.14` changelog entry describing the trimmed alert summary prompts.
packages/security_ai_prompts/kibana/security_ai_prompt/security_ai_prompts-9da0cebb-f56b-4199-b4db-ef79b876d842.json	Update `alertSummary` default prompt for `aiForSoc`.
packages/security_ai_prompts/kibana/security_ai_prompt/security_ai_prompts-7d7cf0c3-7b8e-42a1-878d-668b282247eb.json	Update `alertSummary` default prompt for `ease`.
packages/security_ai_prompts/kibana/security_ai_prompt/security_ai_prompts-26eb85d3-c474-4e48-a918-68b93fa409bb.json	Update `alertSummarySystemPrompt` default prompt for `aiForSoc` (JSON contract guidance retained).
packages/security_ai_prompts/kibana/security_ai_prompt/security_ai_prompts-d8aebe8c-8541-4abf-90b9-7e100b606aec.json	Update `alertSummarySystemPrompt` default prompt for `ease` (JSON contract guidance retained).

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

elastic-vault-github-plugin-prod · 2026-08-24T16:19:47Z

✅ All changelog entries have the correct PR link.

infra-vault-gh-plugin-prod · 2026-08-24T16:40:05Z

💚 Build Succeeded

Buildkite Build
Commit: 0c9b764

mergify · 2026-08-25T13:40:12Z

Queued — the merge queue status continues in this comment ↓.

mergify · 2026-08-25T15:04:32Z

mergify · 2026-08-25T15:04:56Z

queue

☑️ Command `queue` ignored because it is already running from a previous command.

## Summary ### 🛑 Problem `ALERT_ATTACHMENT_PROMPT` and `ALERT_SUMMARY_SYSTEM_PROMPT` had grown into long, step-by-step instruction lists that duplicated guidance already encoded in registered skills like `alert-analysis`. Because skill content is injected into the system context for every session, these verbose prompts were pure overhead — wasted tokens and a second source of truth that would drift out of sync as skills evolved. There was also no eval coverage for either prompt path, so regressions would be invisible. ### 💡 Solution **Prompt changes:** - `ALERT_ATTACHMENT_PROMPT` is now a single sentence that states the user's intent and explicitly delegates the *how* to the `alert-analysis` skill. - `ATTACK_DISCOVERY_ATTACHMENT_PROMPT` is trimmed to the same shape in anticipation of the planned `attack-analysis` skill. - `ALERT_SUMMARY_SYSTEM_PROMPT` is tightened by collapsing redundant "no code fences" rules and merging duplicate markdown-formatting bullets. The `{ summary, recommendedActions }` JSON contract the EASE flyout's `JSON.parse` depends on is preserved verbatim. - `ALERT_SUMMARY_500` is reworded to be more direct while preserving the 500/200 character constraints. **Eval coverage (new):** - `evals/single_alert_triage.spec.ts` — exercises `ALERT_ATTACHMENT_PROMPT` via the Agent Builder `/converse` endpoint with a single alert fixture. Uses an LLM criteria evaluator (markdown format, disposition recommendation, entity grounding) and `createSkillInvocationEvaluator` to verify the `alert-analysis` skill is actually invoked. A single alert does not trigger summary mode, so `expectedAttachmentReads: 0`. - `evals/ease_alert_summary.spec.ts` + `src/ease_summary_task.ts` — exercises `ALERT_SUMMARY_SYSTEM_PROMPT` via the `actions/connector/_execute` route (the path the EASE flyout uses, not Agent Builder). Uses a CODE-only evaluator (`EaseJsonCompliance`) that attempts `JSON.parse` on the raw response and checks that `summary` and `recommendedActions` are non-empty strings. Directly catches the silent fallback-to-raw-text failure mode in `use_alert_summary.tsx`. **Integrations sync:** elastic/integrations#20880 ### 🧠 Notes - The EASE flyout silently falls back to raw text when `JSON.parse` fails, dropping `recommendedActions` with no visible error. The `EaseJsonCompliance` eval is the only gate catching this. Ran locally — scored 1 with the trimmed prompt. - `single_alert_triage` criteria eval also scored 1 locally. - `ease_summary_task.ts` hardcodes the `/internal/elastic_assistant/actions/connector/{id}/_execute` path because `POST_ACTIONS_CONNECTOR_EXECUTE` is not importable from a shared eval package. It explicitly sets `elastic-api-version: 1` as a header — the eval framework's HTTP wrapper does not translate the `version` option the way the browser `HttpHandler` does. - The integrations PR updates only the 4 saved-object files whose content changed. It does not include the UUID→descriptive filename migration that the generator now produces by default; that is a separate concern. ### ✅ How to verify ```bash TRACING_ES_URL=http://localhost:9220 \ EVALUATION_CONNECTOR_ID=<your-connector-id> \ KBN_EVALS_EXECUTOR=phoenix \ KBN_EVALS_SKIP_CONNECTOR_SETUP=true \ node scripts/playwright test \ --config x-pack/solutions/security/packages/kbn-evals-suite-security-alert-triage/playwright.config.ts \ --project <your-connector-id> \ --grep "single_alert|ease" ``` Both new specs should pass with `EaseJsonCompliance: 1` and `criteria: 1`. Alternatively, add the label `evals:security-alert-triage` to trigger the suite in Buildkite. ## Related Integrations sync: elastic/integrations#20880 --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com>

NicholasPeretti and others added 2 commits August 24, 2026 16:56

[security_ai_prompts] Fix changelog link to reference integrations PR

a019596

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

NicholasPeretti requested a review from a team as a code owner August 24, 2026 16:14

Copilot AI lite review requested due to automatic review settings August 24, 2026 16:14

[security_ai_prompts] Point changelog link at replacement PR

0c9b764

elastic#19388 was closed by the stale bot and cannot be reopened, so the changelog link now references the replacement PR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot started reviewing on behalf of NicholasPeretti August 24, 2026 16:15 View session

NicholasPeretti mentioned this pull request Aug 24, 2026

[AI assistant] Align summary prompts elastic/kibana#271330

Merged

Copilot AI reviewed Aug 24, 2026

View reviewed changes

e40pud approved these changes Aug 25, 2026

View reviewed changes

Navigation Menu

Uh oh!

[security_ai_prompts] Sync prompt changes from elastic/kibana#271330 - #20880

[security_ai_prompts] Sync prompt changes from elastic/kibana#271330#20880
NicholasPeretti wants to merge 3 commits into
elastic:mainfrom
NicholasPeretti:sync/security-ai-prompts-kibana-271330

NicholasPeretti commented Aug 24, 2026

Uh oh!

github-actions Bot commented Aug 24, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 24, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Aug 24, 2026

Uh oh!

mergify Bot commented Aug 25, 2026 •

edited

Loading

Uh oh!

mergify Bot commented Aug 25, 2026

Uh oh!

mergify Bot commented Aug 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

NicholasPeretti commented Aug 24, 2026

Summary

Changes

Test plan

Uh oh!

github-actions Bot commented Aug 24, 2026

✅ Elastic Docs Style Checker (Vale)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

elastic-vault-github-plugin-prod Bot commented Aug 24, 2026

Uh oh!

infra-vault-gh-plugin-prod Bot commented Aug 24, 2026

💚 Build Succeeded

Uh oh!

mergify Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergify Bot commented Aug 25, 2026

Merge Queue Status

Uh oh!

mergify Bot commented Aug 25, 2026

☑️ Command queue ignored because it is already running from a previous command.

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mergify Bot commented Aug 25, 2026 •

edited

Loading

☑️ Command `queue` ignored because it is already running from a previous command.