Add Live Debugger runtime benchmark coverage by watson · Pull Request #429 · DataDog/build-plugins

watson · 2026-06-22T13:44:17Z

What and why?

This adds repeatable browser runtime benchmark coverage for dormant Live Debugger instrumentation. The goal is to give Live Debugger browser changes a controlled way to track whether instrumentation changes introduce measurable overhead in real browsers.

How?

Adds a Playwright benchmark under packages/tests/src/bench/liveDebuggerRuntime with Tiny and Hot workload shapes, baseline/control/instrumented variants, and SDK-like dormant probe hooks installed in the page.

Adds a custom reporter that computes conservative per-call overhead bounds, 95% confidence intervals, A/A diagnostics, moving-block bootstrap intervals, autocorrelation diagnostics, browser failures, raw JSON output, and a PR comment body.

Adds yarn workspace @dd/tests bench:live-debugger:runtime, a dedicated Playwright config, result artifacts, a non-blocking CI job, a shared Playwright setup action, and Live Debugger contributor docs explaining how to run and interpret the benchmark.

For more details, see the "Runtime benchmark" section in the added CONTRIBUTING.md file in this PR.

Validation

The benchmark runs in CI as a non-blocking job and uploads raw samples plus the generated report artifacts for review.

Benchmark note

The current benchmark numbers show a significant dormant-instrumentation overhead in Safari. Most of this is fixed in stacked PR #438.

watson · 2026-06-22T13:44:31Z

Add Live Debugger runtime benchmark coverage #429 : 2 dependent PRs (#438 , #440 ) 👈 (View in Graphite)
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

github-actions · 2026-06-22T13:48:12Z

Live Debugger Runtime Benchmark

SDK-loaded dormant-probe runtime overhead, measured against an uninstrumented bundle in the same browser session.

Browser	Workload	Quality	Per-call overhead upper
chrome	Hot	clean	<= 3.38 ns
chrome	Tiny	clean	<= 0.28 ns
firefox	Hot	clean	<= 7.70 ns
firefox	Tiny	clean	<= 0.27 ns
safari	Hot	clean	<= 13.85 ns
safari	Tiny	clean	<= 12.99 ns

Full diagnostics

browser  workload  quality  per-call overhead upper  overhead upper           95% CI        A/A diag         block CI  acf(1)   baseline  instrumented                         samples
-------  --------  -------  -----------------------  --------------  ---------------  --------------  ---------------  ------  ---------  ------------  ------------------------------
chrome   Hot       clean                 <= 3.38 ns        <= 7.82%    3.37..3.38 ns  -0.01..0.01 ns    3.37..3.38 ns   -0.06  51.250 ms     55.255 ms  102 (trim 20%, outliers 11.8%)
chrome   Tiny      clean                 <= 0.28 ns        <= 4.40%    0.23..0.28 ns  -0.01..0.06 ns    0.20..0.28 ns    0.06  53.010 ms     55.140 ms  102 (trim 20%, outliers 18.6%)
firefox  Hot       clean                 <= 7.70 ns       <= 36.15%    7.68..7.70 ns  -0.00..0.01 ns    7.68..7.69 ns    0.02  40.660 ms     55.360 ms  102 (trim 20%, outliers 15.7%)
firefox  Tiny      clean                 <= 0.27 ns        <= 2.27%    0.27..0.27 ns  -0.00..0.00 ns    0.27..0.27 ns    0.08  58.940 ms     60.280 ms   102 (trim 20%, outliers 7.8%)
safari   Hot       clean                <= 13.85 ns       <= 65.86%  13.63..13.80 ns  -0.00..0.00 ns  13.59..13.85 ns    0.02  32.980 ms     54.520 ms   102 (trim 20%, outliers 2.0%)
safari   Tiny      clean                <= 12.99 ns      <= 396.82%  12.74..12.92 ns  -0.00..0.00 ns  12.65..12.99 ns    0.23  12.000 ms     59.170 ms   102 (trim 20%, outliers 2.9%)

Raw samples are in the live-debugger-runtime-bench-results artifact.

datadog-datadog-prod-us1-2 · 2026-06-22T13:50:06Z

✅ All CI checks and tests passed.

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: f3cc0fe | Docs | View more details | Give us feedback!}

The browser Live Debugger instrumentation needs repeatable runtime overhead checks before transform changes land. Add an opt-in Playwright benchmark that compares baseline, control, and instrumented workloads in the same browser session with dormant probe hooks installed. Report conservative per-call overhead bounds with confidence intervals, A/A diagnostics, block bootstrap checks, and PR comment output. Wire the benchmark into CI as a non-blocking job, share Playwright setup between jobs, upload raw samples as artifacts, and document how contributors should run and interpret the benchmark.

Grow near-target batches proportionally and lock the behavior down with a deterministic test. Co-authored-by: Cursor <cursoragent@cursor.com>

tylfin

LGTM, but I'll defer the CI changes and test configuration to the build-plugins team

yoannmoinet

Some comments.

yoannmoinet · 2026-09-03T18:31:15Z

+            packages/tests/test-results
+          retention-days: 3
+
+  runtime-bench-preflight:


Shouldn't this also be continue-on-error: true since the other one is?

yoannmoinet · 2026-09-03T18:32:55Z

+        id: output-check
+        run: node packages/tests/src/bench/liveDebuggerRuntime/preflight.js --compare-output --base-ref=origin/${{ github.base_ref }}
+
+  runtime-bench:


Since we're posting comments in the PRs, maybe we'll need to avoid runs conflicting (quick succession of pushes).

This way, we only keep the latest one running.

Suggested change

runtime-bench:

runtime-bench:

concurrency: { group: runtime-bench-${{ github.event.pull_request.number }}, cancel-in-progress: true }

nit: Should we also configure Test Optimization for this one too?

yoannmoinet · 2026-09-03T18:33:50Z

+    if: ${{ github.event_name == 'pull_request' && !startsWith(github.ref_name, 'mq-working-branch-') }}
+    outputs:
+      should-run: ${{ steps.output-check.outputs.should-run || steps.changed-files.outputs.should-run }}
+      reason: ${{ steps.output-check.outputs.reason || steps.changed-files.outputs.reason }}


nit: unused.

yoannmoinet · 2026-09-03T18:41:03Z

+        key: node18-cache-build-${{ hashFiles('packages/core/**', 'packages/factory/**',
+          'packages/plugins/**', 'packages/published/**',
+          'packages/tools/src/**', 'yarn.lock') }}


Not directly related to this PR, but it does increase an existing race risk.

Three jobs share the same cache key, lint, e2e and now runtime-bench as well.

Although, lint does a build slightly different with ADD_BUILD_PLUGINS and BUILD_PLUGINS_REPORTS.

Which means, if e2e or runtime-bench build first, lint won't have a report to upload.

Could you change the key for lint please?

yoannmoinet · 2026-09-03T18:44:58Z

+    forbidOnly: !!process.env.CI,
+    retries: 0,
+    reporter: [['list'], ['./src/bench/liveDebuggerRuntime/reporter/benchReporter.ts']],
+    globalSetup: require.resolve('./src/_playwright/globalSetup.ts'),


Note that this will conflict with e2e runs, because it cleans the public dirs.
So if this runs at the same time as e2e, it might clear its fixtures mid-run.

Could be unlikely, but I felt like it was worth a mention.

build-plugins/packages/tests/src/_playwright/globalSetup.ts

Lines 37 to 42 in 38eda25

// Delete public dirs.

const cleanPfx = getSubPfx('Clean');

console.time(cleanPfx);

const publicDirs = await glob('public/*/', { cwd: __dirname });

await Promise.all(publicDirs.map((dir) => rm(path.resolve(__dirname, dir))));

console.timeEnd(cleanPfx);

yoannmoinet · 2026-09-03T18:46:27Z

+const BENCH_BUNDLER = 'rspack';
+const BENCH_DEV_SERVER_PORT = 8001;
+const BENCH_DEV_SERVER_URL = `http://localhost:${BENCH_DEV_SERVER_PORT}`;
+const BENCH_BROWSERS = ['chrome', 'firefox', 'safari'] as const;


These could go in ./constants.ts.

watson force-pushed the watson/DEBUG-5787/add-bench branch 5 times, most recently from 3a9f3d6 to 1296572 Compare June 24, 2026 16:05

watson marked this pull request as ready for review June 24, 2026 16:17

watson requested review from a team as code owners June 24, 2026 16:17

This was referenced Jun 24, 2026

Fix Safari dormant overhead in Live Debugger instrumentation #438

Open

Use the real Browser Debugger SDK in the runtime benchmark #440

Open

watson force-pushed the watson/DEBUG-5787/add-bench branch 2 times, most recently from d80cc4f to ba9d257 Compare June 26, 2026 11:49

This was referenced Jun 29, 2026

Fix live debugger curried function instrumentation #441

Draft

Fix Live Debugger returned arrow instrumentation #442

Draft

watson force-pushed the watson/DEBUG-5787/add-bench branch from ba9d257 to 0168305 Compare July 1, 2026 13:03

This was referenced Jul 3, 2026

Fix Live Debugger parenthesized arrow body instrumentation #447

Draft

Add decorator support to Live Debugger instrumentation #448

Draft

watson force-pushed the watson/DEBUG-5787/add-bench branch from 0168305 to 0de9917 Compare August 26, 2026 11:15

Reduce benchmark calibration overshoot

f3cc0fe

Grow near-target batches proportionally and lock the behavior down with a deterministic test. Co-authored-by: Cursor <cursoragent@cursor.com>

tylfin approved these changes Aug 26, 2026

View reviewed changes

yoannmoinet reviewed Sep 3, 2026

View reviewed changes

Navigation Menu

Uh oh!

Add Live Debugger runtime benchmark coverage - #429

Add Live Debugger runtime benchmark coverage#429
watson wants to merge 2 commits into
masterfrom
watson/DEBUG-5787/add-bench

watson commented Jun 22, 2026 •

edited

Loading

Uh oh!

watson commented Jun 22, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 22, 2026 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Jun 22, 2026 •

edited by datadog-prod-us1-4 Bot

Loading

Uh oh!

tylfin left a comment

Uh oh!

yoannmoinet left a comment

Uh oh!

yoannmoinet Sep 3, 2026

Uh oh!

yoannmoinet Sep 3, 2026

Uh oh!

yoannmoinet Sep 3, 2026

Uh oh!

yoannmoinet Sep 3, 2026

Uh oh!

yoannmoinet Sep 3, 2026

Uh oh!

yoannmoinet Sep 3, 2026

Uh oh!

yoannmoinet Sep 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	runtime-bench:
	runtime-bench:
	concurrency: { group: runtime-bench-${{ github.event.pull_request.number }}, cancel-in-progress: true }

	// Delete public dirs.
	const cleanPfx = getSubPfx('Clean');
	console.time(cleanPfx);
	const publicDirs = await glob('public/*/', { cwd: __dirname });
	await Promise.all(publicDirs.map((dir) => rm(path.resolve(__dirname, dir))));
	console.timeEnd(cleanPfx);

Uh oh!

Conversation

watson commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What and why?

How?

Validation

Benchmark note

Uh oh!

watson commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Live Debugger Runtime Benchmark

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Jun 22, 2026 • edited by datadog-prod-us1-4 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tylfin left a comment

Choose a reason for hiding this comment

Uh oh!

yoannmoinet left a comment

Choose a reason for hiding this comment

Uh oh!

yoannmoinet Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

yoannmoinet Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

yoannmoinet Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

yoannmoinet Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

yoannmoinet Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

yoannmoinet Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

yoannmoinet Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

watson commented Jun 22, 2026 •

edited

Loading

watson commented Jun 22, 2026 •

edited

Loading

github-actions Bot commented Jun 22, 2026 •

edited

Loading

datadog-datadog-prod-us1-2 Bot commented Jun 22, 2026 •

edited by datadog-prod-us1-4 Bot

Loading