Add receipt audit exports, retention, and SOAR certificate trust improvements

CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## 0.5.6
+
+- Load standard Linux CA bundles alongside Python trust defaults for SOAR HTTPS connections. Preserve explicit runtime CA overrides and certificate/hostname verification.
+- Report certificate verification and TLS handshake failures separately from connectivity timeouts.
+- Add optional retention for delivered submissions, disabled by default. Remove receipt data in bounded, cluster-coordinated batches when submissions are listed; retain failed requests, audit entries, and duplicate-prevention markers. SOAR events are unaffected.
+
+## 0.5.5
+
+- Add an optional receipt timeline with submission, delivery, and reported SOAR run updates. Keep records without timestamps separate from dated updates.
+- Export receipts as JSON with submitted fields, the form snapshot, and available SOAR activity, including result limits and refresh errors.
+- Export the filtered submissions list across pages as CSV, including delivery details, submitted fields, export identity, and scope. Spreadsheet formula values are escaped.
+
 ## 0.5.4
 
 - Remove unsupported `local` options from lookup searches and correct the builder guidance. Existing searches using them show instructions to remove the option.

DEPLOYMENT.md

@@ -109,7 +109,33 @@ Delivery locks do not expire automatically. If a handler crashes while holding a
 - Preserve form revisions, connection snapshots, unfinished submissions, and active delivery locks during retention cleanup.
 - The catalog is paginated; submission lists show the latest 200 authorized records. KV scans are bounded at 50,000 records.
 - Delivery attempts are limited to 10 per user per minute. These limits are shared across members in a cluster. Lookup searches are limited to 60, receipt activity refreshes to 20, and submission-list status reads to 60 per user per minute, with separate budgets.
-- The app does not run a background retry or retention service. Prune old rate-limit records through your administration process, retaining at least the last 24 hours.
+- The app does not run a background retry or retention scheduler. Optional receipt cleanup runs when submissions are listed, as described below. Prune old rate-limit records through your administration process, retaining at least the last 24 hours.
 - Validate role isolation, credential access, delivery/retry behavior, and, for clusters, member failover in your deployment.
 
 For a Splunk Web CSRF error, sign in again and check that the reverse proxy preserves session cookies, `X-Requested-With`, and `X-Splunk-Form-Key`. For a missing-label error, create the configured label in SOAR or change the form's mapping and publish it. Retrying an existing submission keeps its original label.
+
+## Receipt timeline and audit exports
+
+Switch a receipt from **Grouped** to **Timeline** to see submission, delivery, and SOAR run updates ordered by time. SOAR run timestamps are last updates, not start or completion times. Undated blocks appear after dated entries as **Time not reported**. This is a current snapshot; previous status changes and individual delivery attempts are not reconstructed.
+
+**Export JSON** on a receipt downloads the submitted fields, form definition snapshot, delivery details, available SOAR activity, and timeline. The file records the exporting user, app version, export time, and SOAR refresh time. Missing activity, refresh errors, and truncated result flags remain in the export. The button waits for an active activity refresh; failed reads still allow exporting the receipt and any previously loaded activity.
+
+**Export CSV** on Submissions includes all records in the selected workspace/ownership filter across pages, up to the list's 200 most recent accessible records. It exports delivery details and submitted fields as JSON in an `inputs_json` column; use receipt JSON for SOAR run details. The CSV states its scope and exporting identity. Formula-like spreadsheet values are prefixed with an apostrophe.
+
+Exports use only records already returned by the existing receipt access checks. They are snapshots for audit review, not complete historical or tamper-evident audit logs. They do not change retention or retrieve unbounded SOAR history.
+
+## SOAR certificate trust
+
+SOAR HTTPS requests originate from Splunk's Python runtime on a search head. A trusted certificate in an administrator's browser or in another application's trust store does not configure that runtime. ActionStack loads Python's default CA sources, then supplements them with available standard Linux bundles (Debian/Ubuntu `/etc/ssl/certs/ca-certificates.crt`, RHEL-family `/etc/pki/tls/certs/ca-bundle.crt` and `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem`, plus `/etc/ssl/ca-bundle.pem` and `/etc/ssl/cert.pem`). A previously saved app CA chain is also loaded.
+
+Explicit `SSL_CERT_FILE` or `SSL_CERT_DIR` environment overrides are respected; with either set, the additional Linux bundles are not loaded. The Splunk service account must be able to read the configured trust sources. Trust the CA on every search head, configure SOAR to serve its intermediate certificates, and use a URL whose hostname appears in the certificate. Certificate verification failures now include OpenSSL's bounded verification message; they are distinct from TLS handshake failures and network timeouts.
+
+**Ignore certificate validation** remains opt-in. It disables certificate and hostname checks only for this app's SOAR connection. Updating ActionStack does not enable it or change system trust configuration.
+
+## Delivered submission retention
+
+In **Settings → Delivered submission retention**, choose **Keep indefinitely** (default), or 7, 30, 60, 90, 180, 365, or 730 days, then **Save settings**. Only app administrators can change this app-wide policy. Export required receipts before enabling it: removed fields and receipt details cannot be restored through the app.
+
+Cleanup runs when submissions are listed or refreshed. A shared KV Store slot limits cleanup to one batch of up to 100 records per hour across the search heads. The age is measured from the last delivery update, and only records whose delivery status is `submitted` qualify. Recent records, failed/unconfirmed/pending deliveries, and records with a delivery lock remain untouched. An already running batch uses the policy it started with. Cleanup is not scheduled while the app is idle, so a backlog may take multiple visits and batches.
+
+Each removed receipt is replaced at the same KV key by a minimal marker containing its submission ID, `expired` status, and expiration time. This atomic replacement removes the form, submitted fields, actor, connection snapshot, and payloads without creating a gap that could allow duplicate delivery. Markers are excluded from receipt lists and retained indefinitely; do not delete them as part of routine cleanup. Reusing an expired request's submission key is rejected. SOAR events/artifacts, form versions, and ActionStack audit entries are not deleted. Batch counts are recorded as `submissions.expired` audit events.

README.md

@@ -4,6 +4,10 @@ Build forms in Splunk that submit events to Splunk SOAR.
 
 [Splunkbase](https://splunkbase.splunk.com/app/9812) · [Issues](https://github.com/Shorton88/ActionStack/issues) · [Contributing](CONTRIBUTING.md)
 
+## About this project
+
+ActionStack is an AI-driven project attempting to build something useful for the community. It is developed and maintained on a best-effort basis and is provided "as is," without warranty of any kind. See the [MIT license](LICENSE) for the full terms.
+
 ## Features
 
 - Team workspaces with access controlled by Splunk roles.
@@ -12,6 +16,8 @@ Build forms in Splunk that submit events to Splunk SOAR.
 - Drafts, publishing, version history, cloning, and recoverable form deletion.
 - Configurable SOAR labels, tags, CEF mappings, and approval requirements handled by your playbooks.
 - Paginated submission history with playbook/action status counts, delivery retries, and receipts with custom action names, reported block results, summaries, and data.
+- Optional receipt timeline and JSON receipt / CSV submission exports for audit review.
+- Optional retention for delivered submissions, with SOAR events retained.
 - Light, dark, and system themes.
 
 ## Installation

frontend/src/AutomationActivity.tsx

@@ -2,7 +2,9 @@ import { Fragment, useEffect, useState } from "react";
 import { RefreshCw } from "lucide-react";
 import { RunCounts } from "./RunCounts";
 import { api } from "./api";
-import type { Activity, RunGroup } from "./types";
+import { ReceiptTimeline } from "./ReceiptTimeline";
+import type { ActivitySnapshot } from "./receipt-audit.js";
+import type { Activity, RunGroup, Submission } from "./types";
 
 function Runs({ title, group }: { title: string; group: RunGroup }) {
   return (
@@ -106,24 +108,23 @@ function Runs({ title, group }: { title: string; group: RunGroup }) {
 }
 
 export function AutomationActivity({
-  id,
-  containerId,
-  enabled,
+  submission,
+  onSnapshot,
 }: {
-  id: string;
-  containerId: number | null;
-  enabled: boolean;
+  submission: Submission;
+  onSnapshot: (snapshot: ActivitySnapshot) => void;
 }) {
+  const { id, container_id: containerId } = submission;
+  const enabled = submission.form.mapping.run_automation;
+  const [view, setView] = useState("grouped");
   const [data, setData] = useState<Activity | null>(null);
   const [error, setError] = useState("");
-  const [busy, setBusy] = useState(false);
+  const [busy, setBusy] = useState(Boolean(containerId));
   const [refresh, setRefresh] = useState(0);
   useEffect(() => {
     let stopped = false;
     let timer: ReturnType<typeof setTimeout>;
-    setData(null);
-    setError("");
-    setBusy(false);
+
     async function poll() {
       if (stopped || !containerId) return;
       if (!document.hidden) {
@@ -148,6 +149,10 @@ export function AutomationActivity({
       clearTimeout(timer);
     };
   }, [id, containerId, refresh]);
+  useEffect(
+    () => onSnapshot({ data, error, loading: busy }),
+    [data, error, busy, onSnapshot],
+  );
   return (
     <section
       className="automation-activity"
@@ -170,18 +175,37 @@ export function AutomationActivity({
           : "Automatic execution was disabled for this submission."}{" "}
         Event activity includes manual runs and reruns.
       </p>
+      <div className="filter-tabs" aria-label="Receipt activity view">
+        <button
+          aria-pressed={view === "grouped"}
+          className={view === "grouped" ? "selected" : ""}
+          onClick={() => setView("grouped")}
+        >
+          Grouped
+        </button>
+        <button
+          aria-pressed={view === "timeline"}
+          className={view === "timeline" ? "selected" : ""}
+          onClick={() => setView("timeline")}
+        >
+          Timeline
+        </button>
+      </div>
+      {error && (
+        <p role="status" className="activity-error">
+          Status unavailable: {error}
+          {data
+            ? " Displayed activity is from the last successful refresh."
+            : ""}
+        </p>
+      )}
+      {view === "timeline" && (
+        <ReceiptTimeline submission={submission} activity={data} />
+      )}
       {!containerId ? (
         <p className="muted">Waiting for a SOAR event ID.</p>
       ) : (
         <>
-          {error && (
-            <p role="status" className="activity-error">
-              Status unavailable: {error}
-              {data
-                ? " The results below are from the last successful refresh."
-                : ""}
-            </p>
-          )}
           {!data && !error && (
             <p className="muted">
               {busy
@@ -194,9 +218,15 @@ export function AutomationActivity({
               {data.demo && (
                 <p className="muted">Demo mode does not run playbooks.</p>
               )}
-              <Runs title="Playbooks" group={data.playbooks} />
-              <Runs title="Actions" group={data.actions} />
-              {data.blocks && <Runs title="Other blocks" group={data.blocks} />}
+              {view === "grouped" && (
+                <>
+                  <Runs title="Playbooks" group={data.playbooks} />
+                  <Runs title="Actions" group={data.actions} />
+                  {data.blocks && (
+                    <Runs title="Other blocks" group={data.blocks} />
+                  )}
+                </>
+              )}
               <small className="muted">
                 Last checked {new Date(data.checked_at).toLocaleTimeString()} ·
                 Refreshes every 30 seconds while this receipt is visible.

frontend/src/ReceiptTimeline.tsx

@@ -0,0 +1,79 @@
+import { receiptTimeline } from "./receipt-audit.js";
+import type { Activity, Submission } from "./types";
+
+export function ReceiptTimeline({
+  submission,
+  activity,
+}: {
+  submission: Submission;
+  activity: Activity | null;
+}) {
+  const rows = receiptTimeline(submission, activity);
+  return (
+    <div className="actionstack-receipt-timeline">
+      <p className="muted">
+        Oldest update first. Times show the latest reported updates. Previous
+        status changes and individual delivery attempts are not retained here.
+      </p>
+      {activity &&
+        Object.entries({
+          playbooks: activity.playbooks,
+          actions: activity.actions,
+          blocks: activity.blocks,
+        }).map(
+          ([key, group]) =>
+            group && (
+              <div key={key}>
+                {group.error && (
+                  <p role="status" className="activity-error">
+                    {key}: {group.error}
+                  </p>
+                )}
+                {(group.truncated ||
+                  group.summary_truncated ||
+                  group.summary_error ||
+                  group.notice) && (
+                  <p className="muted">
+                    {key}:{" "}
+                    {group.notice ||
+                      "Some results are limited or unavailable. Use Grouped view for details and SOAR for the complete history."}
+                  </p>
+                )}
+              </div>
+            ),
+        )}
+      <ol aria-label="Receipt timeline">
+        {rows.map((row) => (
+          <li key={row.id}>
+            <small>
+              {row.kind} ·{" "}
+              {row.at ? (
+                <time dateTime={row.at}>
+                  {new Date(row.at).toLocaleString(undefined, {
+                    timeZoneName: "short",
+                  })}
+                </time>
+              ) : (
+                "Time not reported"
+              )}
+            </small>
+            <div>
+              <b>{row.name}</b>
+              <span className={`run-status run-${row.status}`}>
+                {row.status === "submitted"
+                  ? "Delivered"
+                  : row.status === "success"
+                    ? "Succeeded"
+                    : row.status.charAt(0).toUpperCase() +
+                      row.status.slice(1).replaceAll("_", " ")}
+              </span>
+            </div>
+            <small className="actionstack-timeline-reference">
+              {row.detail}
+            </small>
+          </li>
+        ))}
+      </ol>
+    </div>
+  );
+}

frontend/src/connection-settings.js

@@ -10,6 +10,7 @@ export function connectionSettingsPayload(settings, token = "") {
     ca_pem: settings.ca_pem,
     ignore_certificate_errors: settings.ignore_certificate_errors ?? false,
     request_timeout: settings.request_timeout,
+    retention_days: settings.retention_days ?? 0,
     label_prefix: settings.label_prefix ?? "",
     revision: settings.revision,
     ...(token ? { token } : {}),